Building a Better Svelte Data Flow

This blog has been running as a Svelte-powered Sapper app since October 2020. Before changing frameworks from Gatsby I put together a test site and wrote a blog post about what I learned about configuring Sapper. I found a set up that worked for me, so I switched my site over.

The data flow I explored on my test site worked perfectly enough until I ran into a snag — I wanted to compose a post with information from two different sources without having to load data client side and rerender. I built a solution that I think combines the best of Sapper’s native data flow with mdsvex’s powerful preprocessing.

This post is a somewhat deep dive, so if it’s confusing checkout the original Sapper/mdsvex post and come back after that.

If you want to skip the technical explanations of Sapper and mdsvex and get right to how I solved the problem, start here.

How does Sapper work?

Sapper uses file based routing where files and folders are used to build out the page structure of the final site. The Sapper documentation has a section about how this works, but if you’re new to the framework I think it’s more informative to look at the /routes directory in the sapper-template starter repo.

Each file in /routes corresponds to a url endpoint on the site. So /routes/about will produce /about.html, /routes/about/me produces /about/me.html and so on with no limit for how deep nesting goes. This post is mostly about routes that use dynamic parameters, examples of which can be found in the sapper-template’s blog directory.

The sapper-template skips over this step for brevity, but normally a site will contain a collection of markdown files containing the content for each post. In order for Sapper to use this content, two dynamic routes need to exist — index.json.js and [slug].json.js.

index.json.js will use the node file system API to look through the markdown files and build a list. [slug].json.js needs to use something like unified to turn the markdown into valid HTML.

1. blog/[slug].md
2. fs & unified()
1. blog/index.json
2. blog/[slug].json

.svelte files inside the /routes directory get access to a special preload() function that can be called from the <script context='module'> tag. Inside this function pages can access any params captured via the brackets in their file name. [slug].svelte can use params.slug and fetch() to make a call against the corresponding /[slug].json route and get data.

1. browser visit
2. request blog/post
3. blog/post.svelte
4. fetch(`post.json`)
5. blog/post.json
6. preload(){
return data
}
7. blog/post.html

If the app is being run in sapper build mode, [slug].svelte will fetch data and generate a page whenever a user visits a url that corresponds to an existing markdown file. Building a site with sapper export will crawl any links on a site and pre-build and statically export any pages it finds.

1. blog/[slug].md
2. npm run sapper export
3. blog/[slug].hmtl

One final important Sapper concept is layouts. A file named _layout.svelte will automatically be rendered for any page within that route, with the component passed into the _layout’s default <slot />. This is helpful for any elements that will be repeated on every final page, like a header and footer or navigation. One very important consideration, however, is that because of how data flows from parent to child, a _layout does not have access to the $$props object that its child receives.

1. blog/_layout.svelte
2. <slot />
3. blog/[slug].svelte
4. svelte.compile()
5. blog/[slug].html

How does mdsvex work?

mdsvex is a tool that lets you embed functional Svelte components right into Markdown files. In fact, this is how the charts on this page are working. It also hooks into Sapper’s call to svelte.preprocess to take away some of the manual work of creating pages. mdsvex is a preprocessor for Sapper, so all of the same rules for the Sapper /routes directory still apply.

Instead of having to deal with dynamically named [slug].svelte files, mdsvex will take any files with a given extension and create Svelte components. Because these files live within the /routes directory Sapper will see them as pages and create routes for them.

1. blog/[slug].md
2. svelte.preprocess([
mdsvex()
])
3. blog/[slug].svelte
4. npm run sapper export
5. blog/[slug].html

Under the hood, mdsvex fetches data similarly to Sapper, but uses the markdown’s frontmatter to create a data object and pass it into the component as props. This takes the place of manually running fetch inside of the preload function. mdsvex will parse markdown into HTML, and can be given a Svelte layout component that will function similar to a Sapper _layout. Unlike a _layout, an mdsvex layout will be passed the $$props object.

1. blog/post.md
2. mdsvex.compile()
3. {frontmatter}
4. $$props
5. template.svelte
6. <slot />
7. blog/post.svelte

Once these Svelte pages are created, the same flow for Sapper applies. In order for mdsvex to work with Sapper, all scripts need to be run with the --ext flag. This flag will allow Sapper to treat non .svelte files as routes and pick them up during the sapper export process. This is an important distinction — instead of relying on Sapper to crawl and generate each page, mdsvex will always create a Svelte file and corresponding route. Another important caveat here is that mdsvex-generated components will not run a preload function. These genreated components are regular Svelte components and don’t work the same as Sapper page routes.

1. blog/[slug].md
2. npm run sapper export
3. blog/[slug].svelte
4. svelte.compile()
5. blog/[slug].html

Shortcomings

Locking in to a workflow usually comes with a series of tradeoffs. By far the largest benefit of using mdsvex is the ability to put components directly into markdown files. I was able to write over a dozen posts this way before running into any complications. mdsvex is great at taking data from one .md file and transforming it into one .svelte file, but what if data needs to come from two sources?

My real world example of this popped up when I was adding the post-as-a-series feature to my blog. Each series would have a .json file containing a name and short description. Each post in that series contains a frontmatter series field. The posts’ series field would need to correspond to a series name in order to link to extra data. This would function similar to a primary key and foreign key database relationship.

One way to link between this data would be to do the work in the node code that runs to generated each post.json file.

1. blog/post.md
2. blog/series.json
1. fs & unified()
2. blog/post.json
3. npm run sapper export
4. blog/post.svelte

A different way to do this would be to run fetch() against multiple endpoints and return one final data object to the component.

1. blog/post.svelte
2. fetch(`post.json`)
fetch(`series.json`)
1. blog/post.json
2. blog/series.json
3. preload(){
return data
}
4. blog/post.svelte

Because I’m set on using components in my markdown, I have to use mdsvex. It works a little differently as it’s data sources are one-in, one-out. Getting extra data from the blog/series.json route would need to be done inside of an onMount() callback. This would happen moments after a user loads the page and wouldn’t happen during server side prerendering at all. This would trigger a flash of content loading, which I really want to avoid. “Cumulative Layout Shift,” or “CLS” is one of Google’s new Web Vitals, so this would hurt my Lighthouse scores and SEO page rankings.

1. blog/post.svelte
2. render
3. blog/post.html
4. onMount(){
() => fetch()
}
5. blog/series.json
6. render again
7. blog/post.html

Fetching data in other places

The other option is to run fetch() from a Sapper _layout file that wraps each page, but this presents an interesting problem. Because of the way that Svelte “surgically updates the DOM,” the _layout.svelte file is actually mounted and rendered once and its <slot /> content is dynamically changed out when Sapper changes routes.

You can force a refetch of a data here using a reactive statement and pass it down, but because of the way Svelte’s lifecycle works this will result in the same problem where data is loaded after initial render.

Combining both page composition methods

Let’s review what we know at this point —

• Sapper can use a dynamic component to take slug from a url, turn it into a file path, then fetch and preload data.
• fs can make a list of files, then sapper export can crawl it to produce HTML pages.
• mdsvex will generate a component from any file in a directory, even if the routes are not crawled by Sapper.
• In order to get data at the right time, it needs to be fetched from the preload method of a page.
• Anything done in a preload() from a route can be server rendered by sapper export.

So what would an ideal data flow look like?

A browser (or the Sapper crawl process) would visit a url and load a dynamic [slug].svelte route. This route would then be able to call a preload function and fetch any json data, as well as find the .md content that’s been transformed by mdsvex.

1. browser visit
2. request blog/[slug]
3. blog/[slug].svelte
4. preload()
1. blog/[slug].json
2. blog/[slug].md
5. server side render
6. blog/[slug].html

Getting data

This is pretty much the default Sapper workflow, but we need a way to loop in components generated by mdsvex without letting it take over as the de facto route generator. There are three important features that can work together to do this —

• Sapper files and directories with a leading underscore do not create routes.
• mdsvex can import { metadata } from any file’s frontmatter, but it also exports the generated default transformed page body.
• mdsvex will generate these components even if a route is not created.

If a .md file lives directly inside routes/blog, mdsvex is going to turn it into a page and Sapper will load it for a given url instead of the generic [slug].svelte. The first thing to do is avoid this by moving all of the posts into a _content directory.

└─ blog/
  ├─ _content/
  │ ├─ post1/
  │ │ └─ index.md
  │ └─ post2/
  │   └─ index.md
  ├─ [slug].svelte
  └─ _series.json

The next step is to manually find these files from [slug].svelte’s preload function. For this we need to add the @rollup/plugin-dynamic-import-vars package so we can use the slug to find and load the component that mdsvex generates. import statements won’t usually work with dynamic template strings, but @rollup/plugin-dynamic-import-vars will let us import from a file path that includes a slug variable.

With the plugin installed, [slug].svelte can load this file, get the default export containing the .md content and the metadata export containing the frontmatter, and still make a fetch() call to /series.json and sync up the data.

<script context='module'>
  export async function preload(page) {

    const { slug } = page.params

    const component = await import(`./_content/${slug}/index.md`)

    const series = await this.fetch(`/blog/series.json`)
      .then(response => response.json())
      .then(series => logicToLinkUpPostToSeries(...))
    )

    return {
      page: component.default,
      metadata: component.metadata,
      series: series
    }
  }
</script>

Since all of this happens within an asynchronous preload function the page will wait to render until the data is resolved and we can avoid components loading in after initial render.

Rendering the page

There are two things to be aware of for rendering this data.

First, is component.default. If you console.log() the import it will show different two ydrthings depending on where it is logged.

//  console.log(component.default) from <script context='module'> tag
{ render: [Function: render], '$$render': [Function: $$render] }

//  console.log(page) from regular <script> tag
class Post { constructor(options) }

What’s going on here is best explained in the Routing section of the Sapper docs:

When a user first visits the application, they will be served a server-rendered version of the route in question, plus some JavaScript that ‘hydrates’ the page and initialises a client-side router. From that point forward, navigating to other pages is handled entirely on the client for a fast, app-like feel.

In short, this means that the Sapper server will render the component using the render() function to return stringified HTML. Then the Svelte client will hydrate that HTML by instantiating the class at run time in the browser. Instead of sorting this out manually, the entire component can be passed into a special <svelte:component /> tag that will automatically take care of running each at the appropriate time.

1. blog/page.svelte
2. svelte.compile()
1. function render()
2. class Page
3. <svelte:component />
4. client
   hydration

The second concern is getting data to the correct component. Since some of the work is done behind the scenes by compilers, we can’t always imperatively pass props down from one component to its direct children. Svelte provides a way around this with the context API . Rather than having to send a prop directly down the chain from parent to child, context provides a way to skip links in the chain and get data from a parent to any deeply nested child.

<App>
  <_layout>
    <mdsvexLayout>
      <[slug].svelte>
        <svelte:component this={Page} />
      </[slug].svelte>
    </mdsvexLayout>
  </_layout>
</App>

After getting the series.json data inside of [slug].svelte we can call setContext to store data inside of a context object. Since the <Page /> component generated by mdsvex will be a child component we can call getContext and check if the current page has a value for series and the UI can act accordingly.

<script context='module'>
  export async function preload(page) {
    ...
  }
</script>

<script>
  export let page
  export let series
  import { setContext } from 'svelte'
  setContext('series', series)
</script>

<svelte:component this={page} />

// page.svelte
<script>
  import { getContext } from 'svelte'
  const series = getContext('series')
</script>

{#if series}  
  ...
{/if}

The final data flow works like this.

1. blog/[slug].md
2. blog/[series].json
1. npm run export
2. template.svelte
3. <slot />
4. blog/[slug].svelte
5. preload()
1. blog/[slug].json
2. blog/[slug].md
6. server side render
7. blog/[slug].html

My sapper-mdsvex-starter

If this sounds cool or useful to you (and I hope it does!), I’ve put together a starter template repo that can be used to get a blog with this data flow up and running.

This data flow is weird. I couldn’t find anything else online using a similar set up, or even a different tool chain to solve the same problem. This was able to solve my use case, but if there’s a better way or just any feedback, feel free to let me know in the GitHub issues, on Twitter, or on the Svelte discord where I’m @ryfill.

Special thanks

Huge thank you to everyone in the Svelte discord for helping me rubber duck through this code, especially Jacob Babich. Also thank you to kev and pngwn for fact checking this post.