Super Duper Feeder Documentation

Introduction

Super Duper Feeder is a service designed to subscribe to RSS/Atom feeds on your behalf. It utilizes WebSub/PubSubHubbub for real-time updates when available by subscribing to the feed's external hub, or falls back to periodic polling otherwise.

Feed updates received (either via WebSub push or polling) can be delivered directly to your own application via a configured webhook URL.

This documentation provides detailed information about the service's API endpoints, webhook usage, and callback verification process.

WebSub Concepts

WebSub (formerly PubSubHubbub) is a protocol that enables real-time notifications for content updates. It involves three main components:

• Publishers: Content creators who notify the hub when they update their content
• Hub: An intermediary server facilitating communication between publishers and subscribers. Publishers notify the hub of updates, and the hub pushes those updates to subscribers. **Super Duper Feeder acts as a subscriber to these external hubs.**
• Subscribers: Services or applications (like Super Duper Feeder, or the developer's webhook registered with it) that want to receive real-time updates about content changes from a hub.

The flow works as follows:

1. A subscriber (like Super Duper Feeder) discovers the hub URL from a publisher's feed.
2. The subscriber sends a subscription request to the external hub.
3. The hub verifies the subscription request with the subscriber (using the `/callback/:id` endpoint of Super Duper Feeder).
4. When the publisher updates content, they notify their designated hub.
5. The hub fetches the updated content and sends it to all its verified subscribers (including Super Duper Feeder).
6. Super Duper Feeder receives the update and forwards it to any developer-registered webhooks for that feed.

API Reference

GET {your-callback-url}?mode=verify&token={verification-token}

// Example using Express.js
app.get('/my-callback', (req, res) => {
  // Check if this is a verification request
  if (req.query.mode === 'verify' && req.query.token) {
    // Return the token as plain text
    res.type('text/plain').send(req.query.token);
  } else {
    // Handle other requests
    res.status(400).send('Invalid request');
  }
});

{
  "success": true|false,
  "message": "Description of the result",
  "usingExternalHub": true|false,
  "subscriptionId": "uuid-of-subscription",
  "callbackId": "uuid-of-callback", // Only if callback was provided
  "pendingVerification": true|false, // Indicates if the callback needs verification
  "verificationInstructions": "Instructions for verifying the callback" // Only if pendingVerification is true
}

// Using fetch API in JavaScript
async function subscribeToFeed(topicUrl, callbackUrl) {
  const formData = new FormData();
  formData.append('topic', topicUrl);
  
  if (callbackUrl) {
    formData.append('callback', callbackUrl);
  }

  const response = await fetch('https://superduperfeeder.deno.dev/api/webhook', {
    method: 'POST',
    body: formData
  });
  
  const result = await response.json();
  
  if (result.success) {
    console.log(`Successfully subscribed to ${topicUrl}`);
    console.log(`Using external hub: ${result.usingExternalHub}`);
    console.log(`Subscription ID: ${result.subscriptionId}`);
    
    if (result.callbackId) {
      console.log(`Callback ID: ${result.callbackId}`);
    }
  } else {
    console.error(`Failed to subscribe: ${result.message}`);
  }
}

{
  "success": true|false,
  "message": "Description of the result"
}

{
  "success": true,
  "message": "Callback verified successfully"
}

{
  "success": false,
  "message": "Invalid verification token"
}

{
  "success": false,
  "message": "Verification token has expired"
}

Usage Examples

Subscribing via API

Use the /api/webhook endpoint to subscribe Super Duper Feeder to a feed and optionally register your callback URL.

// Using fetch API in JavaScript
async function subscribeToFeed(topicUrl, callbackUrl) {
  const formData = new FormData();
  formData.append('topic', topicUrl);

  if (callbackUrl) {
    formData.append('callback', callbackUrl);
  }

  const response = await fetch('https://superduperfeeder.deno.dev/api/webhook', { // Replace with your instance URL if self-hosting
    method: 'POST',
    body: formData
  });

  const result = await response.json();

  if (result.success) {
    console.log(`Successfully initiated subscription for ${topicUrl}`);
    console.log(`Using external hub: ${result.usingExternalHub}`); // True if WebSub hub found, false if polling
    console.log(`Subscription ID: ${result.subscriptionId}`);

    if (result.callbackId) {
      console.log(`Callback ID: ${result.callbackId}`);
      if(result.pendingVerification) {
        console.log("Callback requires verification. Check your callback endpoint for a GET request with ?mode=verify&token=...");
        console.log("Verification Instructions:", result.verificationInstructions);
      } else {
        console.log("Callback is verified.");
      }
    }
  } else {
    console.error(`Failed to subscribe: ${result.message}`);
  }
}

// Example call:
// subscribeToFeed('https://example.com/feed.xml', 'https://my-app.com/webhook');
// subscribeToFeed('https://another-blog.com/rss'); // Without a callback

Handling Callback Verification

If you provide a callback URL when subscribing via /api/webhook, Super Duper Feeder needs to verify that you control that URL. It will send a GET request like this:

GET {your-callback-url}?mode=verify&token={verification-token}

Your endpoint must respond with a 200 OK status and return the exact {verification-token} value as plain text in the response body.

// Example using Express.js
app.get('/my-webhook', (req, res) => {
  // Check if this is a Super Duper Feeder verification request
  if (req.query.mode === 'verify' && req.query.token) {
    console.log(`Received verification request for token: ${req.query.token}`);
    // Return the token as plain text to verify ownership
    res.type('text/plain').status(200).send(req.query.token);
    return;
  }

  // Handle other requests (like receiving actual feed updates via POST)
  res.status(400).send('Invalid request');
});

Receiving Feed Updates at Your Callback

Once your callback is verified (or if you didn't provide one), Super Duper Feeder will forward feed updates it receives (from external WebSub hubs or its own polling) to your registered callback URL via a POST request. The request body will contain the feed update content.

// Example using Express.js (add this to your '/my-webhook' route)
// Make sure to use appropriate body parsing middleware (e.g., express.raw, express.text, express.json depending on expected feed types)
app.post('/my-webhook', express.raw({ type: '*/*' }), (req, res) => { // Use appropriate type or '*/*'
  const feedUpdateContent = req.body.toString(); // Or parse as needed
  const contentType = req.headers['content-type'];

  console.log(`Received feed update (${contentType}):`);
  console.log(feedUpdateContent);

  // --- Process the feed update content here ---
  // Example: Parse XML, extract items, store in database, etc.


  // Acknowledge receipt to Super Duper Feeder
  res.status(200).send('OK');
});

Important: Your endpoint should respond quickly (e.g., within a few seconds) with a 2xx status code to acknowledge receipt. Perform time-consuming processing asynchronously. Failure to respond promptly might cause Super Duper Feeder to consider the delivery failed and retry.

Best Practices

• Implement callback verification properly

  Always ensure your callback endpoint correctly handles verification requests. This is essential for security and ensures your callback will receive updates.

Implement proper error handling

Add robust error handling for all API calls and webhook processing.

• Implement retry logic

  Add retry logic for failed webhook deliveries to ensure you don't miss updates.

• Ensure quick responses

  Your callback endpoint should respond quickly (within a few seconds) to verification requests and content deliveries with a 2xx status code. Process heavy tasks asynchronously.

Troubleshooting

Common Issues

• Callback verification fails

  If your callback URL is not being verified:

  • Ensure your callback URL is publicly accessible (not localhost or behind a firewall)
  • Check that your endpoint correctly responds to verification requests by returning the token
  • Verify that your endpoint responds with a 200 OK status code
  • Make sure the response body contains only the token, with no extra whitespace or HTML
  • If using a framework, ensure it's not adding any extra content to the response
• Verification token expired

  If you see "Verification token has expired":

  • Tokens expire after 24 hours
  • Submit a new subscription request to get a new verification token
  • Ensure your callback endpoint is set up correctly before requesting a new token
• Not receiving updates

  If your callback is not receiving updates:

  • Check that your callback has been verified (the pendingVerification flag should be false)
  • Ensure your callback endpoint is functioning properly
  • Check that the subscription is active and has not expired
  • Verify that the feed is actually being updated