Infinite Scrolling that Works


#1

Shortly after we began working together on Discourse, Jeff wrote a post about infinite scrolling. At first, I was surprised at how many people claimed to hate sites that used it. However, after reading through many comments about it, I realized that most didn’t hate the scrolling itself, they hated how it broke their browser!

When I visit Twitter, I am presented with a list of tweets in reverse chronological order. If I scroll down far enough, Twitter will automatically load more tweets so I don’t have to stop reading. Initially, their implementation seems great. I can keep scrolling until I’m done reading.

Twitter’s infinite scrolling has a huge limitation – it only works while you keep your tab open. Try this: open a Twitter window, and scroll down a fair bit. Remember the tweet at the top of your browser window. Restart your browser. If you’re using Chrome, it should re-open all your tabs automatically. Is the tweet you remembered anywhere near the top of your screen? No. Personally, when I do this, I get stuck at the bottom of the first set of tweets Twitter loads.

The engineers at Twitter seem aware of this flaw. Their whole UI is designed around making sure you never lose state in that one tab. All links in tweets open in new tabs when clicked. Your private messages load in a pop-up. Clicking on a user shows a modal with their most recent tweets. If you want to see more of their tweets, it reloads the whole tab with a view of just that user.

After you’ve read a few tweets, what if you want to go back to where you left off in your initial stream? Sorry! Your back button only goes back to the top.

How Discourse deals with these issues

Discourse has infinite scrolling, but it doesn’t have any of the aforementioned issues Twitter does. If you scroll down in a topic and close your browser, you’ll end up right back where you left off. We don’t force links to open in new tabs by default: if users want to do that, they can do it themselves.

How do we do this this? By taking advantage of HTML5’s History API.

Compare the URL

You might have noticed that as you scroll through a topic in Discourse that the URL changes. At first, a URL in a topic looks like this:

http://meta.discourse.org/t/is-it-better-for-discourse-to-use-javascript-or-coffeescript/3153

After you’ve scrolled down a little bit, it will look like this:

http://meta.discourse.org/t/is-it-better-for-discourse-to-use-javascript-or-coffeescript/3153/4

The /4 that is appended to the end refers to the current post number you’re looking at in the topic. In this case, your screen is positioned near the top of the 4th post of topic id 3153.

The replaceState function in the History API allows us to do this. replaceState tells the browser that the URL has changed and the new one should be used if the user hits the back button or reopens a closed tab.

To complete the back button functionality, we then have to support incoming URLs with a post number in them, so we can restore the view the user saw before they left. In Discourse, a URL with a post number doesn’t mean “only show me post x”, it actually means “give me a bunch of posts with x near the top.”

The rest of our infinite scrolling implementation is fairly straightforward. When you reach the bottom of the posts in memory, we trigger a call to load posts after the last post we know. If you scroll upwards to the top, we load the posts before the first post.

The URL is the serialized state of your web application

URLs are meant to represent the location of resources - but I often think people are too focused on resources as documents or videos. Why shouldn’t a URL mean “the posts near post 100”? Twitter’s URL is almost always twitter.com, and their user experience suffers for it.

I won’t claim that Discourse’s approach to infinite scrolling is perfect. There is certainly room for improvement and of course we’d love contributions to our source code. However, I do feel we’ve taken the right approach to infinite scrolling so far by persisting user state in the URL as they scroll.


Imported from: http://eviltrout.com/2013/02/16/infinite-scrolling-that-works.html

#2

I truly wish you good luck with ie issues. History API works nicely on other browsers, but it can become quickly a mess if you have to create a whole new implementation for ie. Back button especially causes annoyances, if you choose to go with the #hash history state.


#3

wouldn't it be better from SEO point of view if the the post number was not in path but rather after a hashtag? Because as it is(/4), it could seem for a search engine as if there are multple pages when in fact there is just one. Hmm?


#4

That shouldn't be an issue for them. :) http://www.discourse.org/faq/#...


#5

Keep in mind that while you are claiming that Twitter's approach to infinite scroll is broken because it's not pushing state, they might counter that you've just broken the "back" button.
A compromise might be to use replaceState instead, so as not to completely obliterate a user's browsing history.

also: http://engineering.twitter.com...


#6

So, do I understand it correctly that you remove the DOM nodes for posts too far away? If yes, this is how infinite scrolling should be done. Twitter's tab approaching 2GB in RAM is not amusing, and much less so on mobile devices.


#7

I wish Tumblr implemented this replaceState thing too. Accidentally back button while reading the dashboard == having to scroll and scroll again :(


#8

This is definitely an interesting approach.

How, if at all, do you account for people sharing the link to friends? I'll often read a topic all the way to completion, then copy/paste the URL from the browser's address bar into a chat window. With your replaceState approach, I'd be sending them a link that will be auto-scrolled to the final message in the thread (a pseudo-permalink to the reply) rather than a link that starts at the top so that they can read the thread themselves naturally.

I've already had friends become confused by links that I send them, because they don't know if I'm pointing out the reply or the full thread. Obviously this is solved on my end by removing the trailing "/N" or jumping to the top of the page before copying the link. However, I don't think either of those solutions would be obvious to an average user.


#10

I had the same thoughts about twitter. They are even overriding the default behavior of the browser. If you tried to press F5 the *window* wont reload but instead it will send a request to the server to check if there're any new tweets or not. Try focusing inside the body of the document (not on the address bar for example) and hit F5 to see it.

As you mentioned "Their whole UI is designed around making sure you never lose state in that one tab."

The new HTML5 History API is one of the greatest yet underestimated APIs on the web sadly. It's an essential part of any single page application if you're concerned about usability and bookmarking.


#11

Backbone.js successfully got pushState to fall back on IE using hashbangs. Google 'pushstate fall back to hashbangs on iE' and see the magic


#13

I'm confused - I was just about to ask what implementation Tumblr use because theirs works very well for me. When I return to the steam it's always in the right place.


#14

One problem with this is that it sort of breaks the home key. Using Google Chrome, I went to that "JavaScript or CoffeeScript" page at the URL above. I scrolled down to the very bottom. Then I closed the tab and restored it. At this point, pressing the home key moved me to somewhere in the middle of the page, not to the top as expected (ctrl+home didn't work either). I had to hit the home key a total of 3 times to get to the top. Clicking on "jump to the first post" up-arrow worked fine, though.


#17

I didn't fully understand the approach of "posts near post 100". Wasn't it supposed to use post pages and therefore get a range/page to primarily load as your stream gets reloaded?

I have a problem with the standard scrolling used in web applications, besides the problems you mentioned from the twitter experience, regarding state. So, either you get what twitter serves (no stream state, one loads the page and lost the last seen content), or you get the facebook (albeit buggy) experience, which scrolls you down, and loads page after page after page, until the content you last saw is visible (in facebook, this can take up to 20-30 seconds if you are quite down in your result stream). I can imagine most servers having issues with loading 20 pages sequentially just to make you go back to your last seen state, which kind of validates what twitter does from a resource-balancing point of view. I see this as us developers not learning the advantages of earlier paradigms like regular pagination (there's one page and one page only loaded in the document) and applying it to the newest "vertical auto-updated resource stream". This is something I like from implementations of such vertical streams in native applications, where there is an event on top called "pull to refresh", which I haven't seen an equivalent and effective system for browser-based applications.

A good way to combine statefulness, resource-loading and user-experience is arguably to load the last page (or the group where post X is included), and then include a button (or something completely new resembling the "pull-to-refresh" event from mobile native apps) on top of the stream which would then either load everything til the most recent post or just the previous page. This approach obviously counts with the user being already down in the stream and having interest in continuing to scroll-down, the most resounding disadvantage being completely losing track of the most updated state (like, if new posts have been added to the top of the stream). This could be mitigated with a notification asking the user to explicitly go back to the first page to check the latest updates.

My opinion is that the autoloading stream, although a nice idea, has been an implementation nightmare ever since its inception. I haven't seen as of yet a good implementation of it combining these 4 characteristics: keeping the last seen state, being aware of updates and notify them properly, balanced resource-loading approach on page refresh, and relevant user experience leveraging the last 3 points. I'd focus on those next time I would have to implement such a solution from scratch.


#19

The other thing that sucks about infinite scrolling is it breaks the browser's scroll bar. Open up a site with infinite scrolling, start dragging the scroll bar to browse down, and the second it loads more content, since the page is now longer, the scroll-bar repositions and jumps you all over the page.

Infinite scrolling only works good if you use the mouse wheel or click-mouse-wheel and use the scroll thing.


#20

Resurrection!!

First of all, thank you so much for discourse and all the amazing job done there Robin !

Now, couple questions:

What happened with your second message there?

It looks like another person talking to you, but coming from your profile, lol.

Secondly, completely agree with @spam there. I believe this could be fixed if the page loaded the proper full page height without any content. I suppose all needed for this is a server side cached rendering of the page and adjust it to the window width.

Had you thought about going this route, @eviltrout? I’m willing to bet it was just not a priority! :slight_smile:


#21

This is not simple at all to implement. In fact I bet it would be a nightmare!


#22

You’re right, I did imply it was simple, and it really isn’t. I never thought it would be easy, though.

But you didn’t answer if this is a new idea or something you had thought about!

Because maybe there are better ways to approach this, or maybe it’s just a waste of time trying to do it.

Being such an important part of discourse, I’m pretty sure you do give a lot of thought on how to improve the endless scrolling. Right now I’m trying to implement a simple one myself, and wondering about all that stuff.

The more I think about it, the more I have to agree. There is just no solution to this, is there?


#23

I hadn’t thought about this approach as it doesn’t really fit our use case at Discourse. So far our scrolling is working fairly well for our use case. There are rough edges but overall it’s good.


#24

I’d argue that as long as the scrolling bar is still there, it sure applies. Also, if this worked you could remove most functions from the current scroll widget or even remove it entirely. Home and End buttons would work out of the box.