by Ben Ubois
Todd and I had the pleasure of working with Dan Cederholm on a new logo for Feedbin.
I’ve been a long time fan of Dan’s design and writing, so when I came across this pitch perfect one-pager for his spiffy Super Turbo Logo Service™ I jumped at the chance.
The process was fun, collaborative, and remarkably smooth. I’m thrilled with the result!
by Ben Ubois
You can now view newsletters with the original formatting. Previously, Feedbin was only able to offer the newsletter content with all styles removed to prevent conflicts between Feedbin’s styles and the newsletter’s styles.
Next up, there are some new management options available in Settings.
When you unsubscribe from a newsletter, future deliveries are blocked to guarantee you don’t see emails that are no longer wanted. However, this creates an issue: what if you want to resubscribe?
To help with this, Feedbin maintains a record of all previous newsletter senders. Deactivated senders are newsletters that you are no longer subscribed to. If you’re trying to resubscribe to a newsletter, you can reactivate that sender.
by Ben Ubois
There’s a new palette to change the theme and other formatting preferences.
The new Auto option will change the theme based on your system preference.
This also comes with a change in behavior: all formatting options are now per-device. This way you can have a light theme on your day phone and a dark theme on your night phone ;) Or use a different font size depending on if you are on a desktop computer or a mobile one.
by Ben Ubois
This Sunday, September 29, Feedbin will be moving to a new hosting environment. The move will mean some downtime. If all goes well, it should be about 30 minutes.
This project has been in the works since April. It’s been a lot of fun and very educational to work on. There will be more posts about the servers and setup soon.
by Ben Ubois
NetNewsWire 5 has been released. I’m thrilled that it:
NetNewsWire was my first introduction to RSS. I’d seen links to RSS feeds around on various blogs, but it took NetNewsWire to put a great interface on the technology and unlock its power. Without NetNewsWire, there would be no Feedbin. It’s a dream come true to be a part of it.
Congratulations and thanks to Brent Simmons, Maurice Parker (who implemented Feedbin sync), and the rest of the NetNewsWire team.
by Ben Ubois
Feedbin now has a read later feature. This enables you to send articles and webpages from anywhere and have them appear alongside your feeds, email newsletters and Twitter subscriptions. It’s called Pages.
There are two ways to get started:
Pages works like a regular feed, so anything that gets sent to it will sync to any client you use with Feedbin.
Finally, Pages helps preserve the content you send to it. Whenever possible, Feedbin will download any images it can find in the content. That way if the images are ever moved or removed, Feedbin can automatically serve the archived images.
by Ben Ubois
The official Feedbin app for iPhone and iPad is now available.
This is a hybrid app, which enables it to achieve full feature-parity with the website. The app allows for better integration and performance than using Feedbin on the web.
The app also lays the groundwork for a great new feature that is coming soon.
by Ben Ubois
There’s an update out today that changes and improves a few things about how Feedbin works. There’s a visual refresh as well. The idea is three unbounded columns that appear to scroll indefinitely.
Universal edit button. Each source now has a consistent editing interface. You can use edit for renaming, tagging, unsubscribing and deleting. Just about everything you can do to a source can now be accomplished using the edit button. The new edit interface makes everything easily accessible on mobile as well.
Grouped sections. The new section headers for Searches, Tags and Feeds help differentiate the types of sources. This helps to reduce clutter as the number of sources in Feedbin increases.
View mode. The control to switch between Unread, Starred, and All view modes is now a dropdown list. This change makes it very clear what you are viewing at all times.
There’s an improvement in behavior here as well. When switching between modes, your currently selected feed will be updated to reflect the chosen mode. For example if you’re viewing a tag called Favorites under Unread and you switch to Starred, the middle column will immediately update to show just the starred articles under Favorites.
by Ben Ubois
Feedbin has a problem. Its articles are stored in a PostgreSQL database, with a 32-bit primary key, that is about to overflow. The largest number you can store in 32 bits is 2,147,483,647 and Feedbin currently is up to id number 2,107,994,090 as of this moment. So there are only about 39 million ids left. At that point no new articles will be able to be created.
Feedbin ended up in this situation because
Luckily, this isn’t a surprise, it’s actually something I’ve been thinking about for almost five years. In November of 2014, Feedbin hit this limit with in the table that stores the unread status of articles. That was a surprise.
However, it also wasn’t a big deal. That table did not need a primary key. The rows in that table could already be uniquely identified with a combined index of user_id + entry_id, so the fix there was to drop the column.
For the entries table, it is a big deal, and dropping this column is not an option.
There are two options:
bigint.Number 2 isn’t a great option. It would require many code changes, other schema changes and for clients that sync with Feedbin to be able to handle UUIDs.
There were a few different ways I identified to switch to a bigint.
ALTER TABLE entries ALTER COLUMN id TYPE bigit;. This one is easy, but it would mean a lot of downtime. This takes out an ACCESS EXCLUSIVE lock on the table, meaning nothing can be written or even read while this runs. It also takes a long time.That brings me to option three. It turns out that I put the problem off long enough that it has been solved for me by a feature added in PostgreSQL 10 last year: logical replication. Previously, Postgres only had physical byte-for-byte replication. Logical replication offers more flexibility because it replays SQL statements like INSERT, UPDATE and DELETE on your replicas. Most importantly for this use-case is that the underlying data type of the replica does not matter as long as the data will still fit into the column.
Here’s how to do this. Let’s say there are two PostgreSQL installations on the same server. Both version 10 and 11 are installed on ports 5432 and 5433 respectively.
Make sure the replica is allowed to connect to the master database
cat >> /etc/postgresql/10/main/pg_hba.conf <<EOL
host all replication 123.123.123.123/32 md5
EOL
Set some configuration options required for physical replication. Also going to temporarily increase the min_wal_size to speed up the transfer.
cat >> /etc/postgresql/10/main/postgresql.conf <<EOL
wal_level = logical
max_wal_senders = 16
max_replication_slots = 8
min_wal_size = 1GB
max_wal_size = 2GB
EOL
Update the configuration on the replica
cat >> /etc/postgresql/11/main/postgresql.conf <<EOL
max_logical_replication_workers = 8
max_worker_processes = 16
max_sync_workers_per_subscription = 6
EOL
Restart the databases so that the new configuration takes effect.
sudo service postgresql@10-main restart
sudo service postgresql@11-main restart
Dump the schema from the master, and change all integer types to bigint.
pg_dump --port 5432 --dbname feedbin --schema-only > schema.sql
sed --in-place 's/integer/bigint/g' schema.sql
Create the replication role and the PUBLICATION on the master.
psql --port 5432 --command "CREATE ROLE replication WITH LOGIN PASSWORD 'password' REPLICATION;"
psql --port 5432 --dbname feedbin --command "CREATE PUBLICATION publication1 FOR ALL TABLES;"
psql --port 5432 --dbname feedbin --command "GRANT SELECT ON ALL TABLES IN SCHEMA public to replication;"
Create the database, role and import the schema on the replica.
psql --port 5433 --command "CREATE DATABASE feedbin;"
psql --port 5433 --command "CREATE ROLE feedbin WITH LOGIN PASSWORD 'password' SUPERUSER;"
psql --port 5433 --dbname feedbin --file schema.sql
Create the subscription on the replica.
psql --port 5433 --dbname feedbin --command "CREATE SUBSCRIPTION subscription1 CONNECTION 'host=localhost dbname=feedbin user=replication port=5432 password=password' PUBLICATION publication1;"
Now we wait. Once the replica is caught up, there’s one more very important part.
Make sure nothing is getting written to the database. Then check the lag until it reaches 0.
psql --port 5432 --command "SELECT application_name, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) lag FROM pg_stat_replication;
The output should look like this when everything is caught up:
-[ RECORD 1 ]----+-----
application_name | subscription1
lag | 0
Logical replication does not bring over sequence data like you would use for auto-incrementing primary keys. We have to copy it over as a separate step.
sequences=$(psql --tuples-only --no-align --quiet --no-psqlrc --port 5432 --dbname feedbin --command "SELECT pg_class.relname FROM pg_class WHERE pg_class.relkind = 'S';")
while read -r sequence; do
count=$(psql --tuples-only --no-align --quiet --no-psqlrc --port 5432 --dbname feedbin --command "SELECT last_value FROM ${sequence};")
echo "SELECT setval('${sequence}', ${count});" >> sequences.sql
done <<< "${sequences}"
This creates the file with sequences.sql which contains a statement for each sequence defined in the database:
SELECT setval('entries_id_seq', 2107994090);
Import the sequence data into the replica.
psql --port 5433 --dbname feedbin --file sequences.sql
Finally update your application to point to the new database. That’s it!
by Ben Ubois
The Mercury Parser API, made by Postlight, is shutting down.
Mercury Parser is the service that powers a number of popular features on Feedbin. These include: extracting the full content from partial content feeds, viewing the content of links in Feedbin, and displaying articles that are linked to from tweets.
However, this is actually good news, because Postlight open-sourced Mercury Parser, and it has already improved significantly. Bugs have been fixed, results have become more accurate, and in the case of Feedbin, it is now much faster.
Feedbin has been using the open-source Mercury project since mid-February, but running on different infrastructure. Postlight also open-sourced the AWS lambda API as a separate project, but I had different requirements for authentication, so I created a small web service to handle authentication and run Mercury Parser.
I was pleasantly surprised to see that running the same service on different hardware boosted performance. This isn’t even dedicated hardware like what Feedbin uses for its primary servers. Just a cluster of Digital Ocean virtual private servers.
Lambda’s infinite scalability sounds very impressive, but infinite scalability is not a requirement for this use-case. Lambda would probably be cheaper (maybe free?), but I’ll always prefer performance at any cost.
Self-hosting also means that I can open up access to this service to Feedbin customers and app developers.
This service is now being used by Reeder to power its content extraction functionality and I want to offer the same arrangement to anyone that makes an app that supports Feedbin. Please get in touch if you’d like to use it in your app, whether your users are logged in to a Feedbin account or not.
For everyone else, the Feedbin entries API now includes an extracted_content_url key. Visiting this URL will return the extracted content of the entry.
Thanks to Postlight for running Mercury Parser free-of-charge all these years! If you know of a site that could use improved extracted content results, you can contribute a custom parser to the project. It’s a straightforward process if you have CSS and JavaScript experience.
P.S. Feedbin turned six today. Happy birthday! 🎂