<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>scambier.xyz — Blog</title>
    <link>https://scambier.xyz</link>
    <description>Blog posts from scambier.xyz</description>
    <atom:link href="https://scambier.xyz/blog.xml" rel="self" type="application/rss+xml"/>
    <generator>Quartz -- quartz.jzhao.xyz</generator>
    <item>
    <title>Userscripts</title>
    <link>https://scambier.xyz/blog/userscripts</link>
    <guid>https://scambier.xyz/blog/userscripts</guid>
    <description><![CDATA[ In a world where technology becomes more utterly fucked every day, userscripts can feel liberating. You don’t know what are userscripts? They’re tiny (or not) javascript scripts that you can inject into any web page. Kinda like a browser plugin, but usually hyper-focused to change specific behaviors or visuals on specific sites.
They’re a great way to take back a bit of control.
You can use them by installing a userscripts manager like Tampermonkey (it’s the one I use).

Now what about a few examples?
Reddit
Reddit is a mess (to put it gently), you know your health would be better if you stopped interacting with it, but it’s still a useful resource. Rejoice, here’s a script for old.reddit.com to disable the reply button and that useless notification icon!
// ==UserScript==
// @name         Better reddit
// @version      2025-07-10
// @description  Less annoyances and lower blood pressure
// @author       scambier.xyz
// @match        www.reddit.com/*
// @match        old.reddit.com/*
// @icon         www.reddit.com/favicon.ico
// @grant        none
// ==/UserScript==
 
(function() {
    &#039;use strict&#039;;
    setTimeout(() =&gt; {
        // You don&#039;t care about notifications
        $(&#039;#notifications&#039;).next(&#039;.separator&#039;).remove()
        $(&#039;#notifications&#039;).remove()
        $(&#039;.message-count.badge-count&#039;).remove()
 
        // Read-only mode
        $(&#039;li.reply-button&#039;).remove()
        $(&#039;div.usertext-edit&#039;).remove()
    }, 100)
})();
itch.io
You like to browse itch.io for quirky games, assets, or inspiration, but you can’t stand the neverending horror and visual novel slops when you’re browsing the site? This script will automatically filter out the horror tag, and dynamically remove visual novels as you scroll down.
// ==UserScript==
// @name         Filter games on itch.io
// @version      2024-06-24
// @description  Filter out games with a specific tag on itch.io
// @author       scambier.xyz
// @match        itch.io/games
// @match        itch.io/games*
// @match        itch.io/games/*
// @icon         www.google.com/s2/favicons
// @grant        none
// @license      MIT
// ==/UserScript==
 
(function() {
    &#039;use strict&#039;;
 
    // Filtering tags only works one tag at a time
    const url = new URL(window.location.href);
    const exclude = url.searchParams.get(&quot;exclude&quot;);
    if (!exclude) {
        url.searchParams.set(&quot;exclude&quot;, &quot;tg.horror&quot;);
        window.location.href = url.href;
    }
 
 
    const EXCLUDED_GENRES = [
        &#039;visual novel&#039;,
    ];
 
    const removeExcludedGenres = () =&gt; {
        $(&#039;div.game_genre&#039;).filter(function() {
            const genreText = $(this).text().toLowerCase();
            return EXCLUDED_GENRES.some(genre =&gt; genreText.includes(genre));
        }).closest(&#039;div.game_cell&#039;).remove();
    };
 
    // Observer to also delete when scrolling and loading new games
    const observeChanges = () =&gt; {
        new MutationObserver(removeExcludedGenres).observe(document.body, {
            childList: true,
            subtree: true
        });
    };
 
    removeExcludedGenres();
    observeChanges();
})();
Hackernews
The “orange hell site” is terrible and I don’t visit often it anymore, but I like this snippet: it adds icons to quickly see the links with the most votes or comments.

&quot;use strict&quot;;
/* globals jQuery, $, waitForKeyElements */
// ==UserScript==
// @name         Hacker News - Most upvoted &amp; most commented links
// @namespace    github.com/scambier/userscripts
// @downloadURL  github.com/scambier/userscripts/raw/refs/heads/master/dist/hackernews.user.js
// @updateURL    github.com/scambier/userscripts/raw/refs/heads/master/dist/hackernews.user.js
// @author       scambier.xyz
// @version      0.0.10
// @description  Show top 🔥👄 links of Hacker News
// @license      ISC
// @require      cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.js
// @match        news.ycombinator.com/
// @match        news.ycombinator.com/ask*
// @match        news.ycombinator.com/news*
// @match        news.ycombinator.com/show*
// @match        news.ycombinator.com/front*
// @grant        none
// @run-at       document-end
// ==/UserScript==
(() =&gt; {
    // Firefox mobile fix, do nothing if icons are already injected
    if ([...$(&#039;[data-userscript=&quot;scambier&quot;]&#039;)].length)
        return;
    const rows = [...$(&quot;tr.athing&quot;)];
    // Select lines
    const items = rows
        // Filter out ads
        .filter((tr) =&gt; $(tr).find(&quot;td.votelinks&quot;).length)
        // Get id and score
        .map((tr) =&gt; {
        var _a, _b;
        return {
            id: $(tr).attr(&quot;id&quot;),
            score: Number((_a = $(tr).next().find(&quot;.score&quot;)[0].textContent) === null || _a === void 0 ? void 0 : _a.split(&quot; &quot;)[0]),
            // Weirdly, .split(&#039; &#039;) does not work
            comments: Number((_b = $(tr)
                .next()
                .find(&#039;a:contains(&quot;comments&quot;)&#039;)
                .text()
                .split(&quot;comments&quot;)[0]
                .trim()) !== null &amp;&amp; _b !== void 0 ? _b : 0),
        };
    });
    // Top 10% for new entries, top 20% for other pages
    const ratio = location.href.includes(&quot;news.ycombinator.com/newest&quot;)
        ? 0.1
        : 0.2;
    // Inject icons
    items.forEach((o) =&gt; {
        const title = $(`tr#${o.id}`).find(&quot;span.titleline&quot;);
        if (getTop(items, &quot;comments&quot;, ratio).includes(o) &amp;&amp; o.comments &gt; 0) {
            title.before(&#039;&lt;span title=&quot;Most commented&quot; data-userscript=&quot;scambier&quot;&gt;👄 &lt;/span&gt;&#039;);
        }
        if (getTop(items, &quot;score&quot;, ratio).includes(o) &amp;&amp; o.score &gt; 0) {
            title.before(&#039;&lt;span title=&quot;Most upvoted&quot; data-userscript=&quot;scambier&quot;&gt;🔥 &lt;/span&gt;&#039;);
        }
    });
    /**
     * Get [ratio] best [items], ordered by [key]
     *
     * @param items
     * @param key
     * @param ratio
     */
    function getTop(items, key, ratio) {
        const count = rows.length;
        return [...items].sort((a, b) =&gt; b[key] - a[key]).slice(0, count * ratio);
    }
})(); ]]></description>
    <pubDate>Tue, 21 Jul 2026 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>Games I&#039;ve played in 2025</title>
    <link>https://scambier.xyz/blog/games-i've-played-in-2025</link>
    <guid>https://scambier.xyz/blog/games-i've-played-in-2025</guid>
    <description><![CDATA[ The Slormancer

This game immediately went to the top of my wishlist as soon as I learned about it’s existence, and it (mostly) delivered. I consider it one of my GOTY 2025.
It’s an ARPG hack’n’slash (aka Diablo-like) that is much deeper than it seems. All weapons have unique build-defining effects, all gear can be crafted to perfection, there is a ton of levels to grind everywhere, and it’s a pure “numbers go brr” kind of game. I only regret the lack of variety in environments, a grind that becomes too tedious into endgame, and a perfectible gamepad implementation.
Still, there’s an insane amount of content for a very modest price.


Escape from Duckov

2nd GOTY, a top-down PvE extraction shooter. If you’re unfamiliar with the genre, the gameplay loop is simple: drop on a large-ish map, complete quests, loot gear and materials, and extract alive… or lose everything. This is a type of game that is predominantly dominated with PvPvE hardcore games (Tarkov, Hunt Showdown), so a lighter, more casual take with ducks was all I needed to have fun.
The difficulty curve is excellent, the maps are full of places to explore, the quests take you everywhere, and you really feel the power progression as a player. There are also already many neat mods. I love this game.
Small caveat: I don’t recommend it on the Steam Deck, unless you’re ok with aiming with the trackpad.


ARC Raiders

Third and last GOTY, the PvPvE extraction shooter for players that don’t like PvP.
I bought it, got killed by some random German streamer, refunded it (because I don’t have enough time to learn to git gud at PvP games), and then bought it again to play with a friend because I really wanted to like this game. It’s casual enough for me, the PvP vs PvE balance makes it not frustrating to lose a game, and the solo lobbies are mostly chill. And then we play duo or trio to burn through our stash and just have dumb PvP fun.
Basic weapons are viable in PvP, the PvE is actually engaging (and you need it for crafting), and it’s just an all-around good game. Avoid their stupid and toxic subreddit, though.


Mouthwashing

As a walking sim, it was “ok”. Thing is, I certainly lack media literacy to fully appreciate it, as I usually just understand games at surface level. I read a few posts about it after finishing it, and I was like “oooooh”, like there are 4 layers deep to each chapter and each action of each character. So it’s probably good if you take your time to analyze and understand all the metaphors.


Deep Rock Galactic: Survivor

Yet another survivors-like, though this one is pretty to look at, and quite difficult.
DRGS has a strong emphasis on crowd control: you’re not going to clear screens of mobs, you’re here to collect minerals (permanent unlocks) and gain xp points to level up your stats and weapon perks (reset after each mission), and have a chance to kill the big boss as quickly as possible.
The only negative point I have is about the loot: it adds a sort of arpg-like layer to the game for the min-maxers, but I’ve never, ever bothered to look at anything I’ve ever looted. IMO it doesn’t fit the genre: I don’t want to look at stats and sort my stash, I just want to play a short brainless game. Fortunately, there’s a button to automatically fit the best gear available.
Anyway, it’s an excellent game for the Steam Deck, with many weapons to unlock and master, and 300 achievements. Dopamine goes brr.


R.E.P.O.

Funniest multiplayer coop game I’ve played in… maybe forever. I wouldn’t play it solo at all though, because it unlocks its full potential if you manage to gather 3+ friends.
The proximity voice chat and design of the robots add A LOT to the wackiness and humor. The way their eyes focus on the closest point of interest, their “south-park-canadian” mouths, the flailing arms… The gameplay loop itself is just an excuse to put yourself in stupidly dangerous situations and laugh at your unfortunate beheaded teammates.
 ]]></description>
    <pubDate>Thu, 25 Dec 2025 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>Collisions in Gamedev - Part 2 - Dead Cells</title>
    <link>https://scambier.xyz/blog/collisions-in-gamedev---part-2---dead-cells</link>
    <guid>https://scambier.xyz/blog/collisions-in-gamedev---part-2---dead-cells</guid>
    <description><![CDATA[ 

                  
                  NOTE
                  
                

This article is part of a series:

Collisions in Gamedev - Part 1 - Simple AABB
Collisions in Gamedev - Part 2 - Dead Cells (you are here)



Ok, this title is a bit presumptuous, but…
The algorithm we’ll see in this article comes from Sébastien “Deepnight” Bénard, one of the original developers of Dead Cells, and from his own account, the game “uses this exact base engine”. You can read about it on his website.

Code available here
This demo feels very different than the AABB we studied in the first article. Here, the entity is not a hard box hitting walls, but feels more slippery, and appears to be gliding around corners.
If you jump right next to the double-block ledge and press right, it will sort of teleport on top of it.
 
How does it work?
I purposefully represented the player as a sort of cross:

The yellow point in the middle is the actual position (the sprite is drawn with a -4,-4 offset)
The red lines are the collision margins
The green extremities are the “allowed” offsets that extend beyond the margins, to the borders of our 8x8 sprite

In this method, entities are represented by a single point. The coordinates have an integer part (the cell) and a float part (the offset inside the cell). An entity is considered to be colliding when the point’s offset inside its current cell moving is too close to a wall.
For exemple, to check a collision on the right:
-- Collision right
if hasCollision(self.cx + 1, self.cy) and self.xr &gt;= 0.75 then
  self.xr = 0.75
  self.dx = 0
end
If the center point offset horizontal value is going over 75% of the cell’s width (6 pixels from the left) , we’re colliding.

And to check a collision on the left:
-- Collision left
if hasCollision(self.cx - 1, self.cy) and self.xr &lt;= 0.25 then
  self.xr = 0.25
  self.dx = 0
end
If the center point offset horizontal value is going under 25% of the cell’s width (2 pixels from the left) , we’re colliding.

Same goes for the top collision (37.5%, 3 pixels from the top), and the bottom collision (50%, 4 pixels from the top).
Collision Response
Like in the original AABB collision code, we don’t have much of a response: in case of a collision, we just reset the offset to its threshold value. But when going over 1px/frame, instead of stopping before the collision, the entity actually enters the wall and then is being pushed back, with a slight bounce-like effect.
What is this method good for?
It’s obviously good enough for Dead Cells, which has a more dynamic gameplay (the character grabs and climbs ledges) than more rigid games like Mario or Celeste.
And since one of the main advantage of this method is that you can more easily navigate around corners, that makes it a good fit, in my opinion, for top-down games.
What works less well
This method, as originally described, has a few downsides and hard limitations:

If you look too close or move too slowly, the teleport effect when gliding around corners can be obvious
The fact that we’re only colliding from the center point of the entity can cause issues in a platformer requiring precise movements.
The bounce effect when going too fast can be visually problematic
Like with AABB, you can clip through a wall when going diagonally
More importantly, our entities cannot be larger than the size of single cell

Can we improve it?
By itself, I think this can be an interesting and “good enough” method as it is, as long as you accept its limitations. The diagonal clipping can easily be avoided with additional checks, like with AABB.
If you wish to improve this method, it’s going to be harder than just tweak the response, and will require case-by-case solutions for each problem, and each specific need of your game. As always, there’s not a one-size-fits-all solution.
For exemple, for down collisions, it’s quite easy to check for a range instead of a point. This allows the player to walk over the ledge without falling
-- Collision down
local cxLeft = self.cx
local cxRight = self.cx
 
if self.xr &lt; 0.25 then
  cxLeft = self.cx - 1
elseif self.xr &gt; 0.875 then
  cxRight = self.cx + 1
end
 
if (hasCollision(cxLeft, self.cy + 1) or hasCollision(cxRight, self.cy + 1))
	and self.yr &gt;= 0.5 then
  self.yr = 0.5
  self.dy = 0
end


Code available here ]]></description>
    <pubDate>Sat, 12 Apr 2025 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>Collisions in Gamedev - Part 1 - Simple AABB</title>
    <link>https://scambier.xyz/blog/collisions-in-gamedev---part-1---simple-aabb</link>
    <guid>https://scambier.xyz/blog/collisions-in-gamedev---part-1---simple-aabb</guid>
    <description><![CDATA[ 

                  
                  NOTE
                  
                

This article is part of a series:

Collisions in Gamedev - Part 1 - Simple AABB (you are here)
Collisions in Gamedev - Part 2 - Dead Cells



Introduction
In this series of articles, I’ll use PICO-8 to implement different collision detection methods, in a basic, straightforward, and interactable way.


                  
                  Not a tutorial 
                  
                

Most of the methods that I plan to showcase already have several free tutorials readily available for you to read and watch. My plan is to succinctly show how a method works, what are the issues and downsides, and how to solve them if possible.
I’ll assume the reader has basic knowledge of game programming and concepts.




                  
                  PICO-8 API 
                  
                

PICO-8 uses a special flavor of Lua, with some specific API functions. So, the examples will obviously use these functions (e.g. flr() instead of math.floor()), but I’ll try to refrain using P8 idioms (e.g. +=) that don’t exist in standard Lua.


Axis-Aligned Bounding Boxes
Or AABB, is a fancy wording to say “rectangles on a grid”. AABB collision detection is the most basic way to check if 2 rectangles (or points, or circles) collide, and it’s also the easiest to get wrong.
Let’s try this minimal demo, we’ll see what’s wrong with it right below. Use the left/right arrows to move, and the up arrow to (repeatedly) jump. Press ctrl+r if you’re stuck.
The code is available here.

How does it work?
For each frame, depending on our direction, we check the next position of the left/right/up/bottom corners. If that next position collides, we stop the entity. Simple, right? Nope.
Wait, it’s actually buggy?
If you’ve played a bit with my demo, you probably noticed a few issues.
#1: If you jump with a block above you, the jump is stopped  before you reach the block
Since your entity is blocked before the collision happens, if you’re moving faster than 1px/frame, your entity will be blocked when there’s still a gap between it and the wall.
#2: There is sometimes a weird “lag” that happens right before the entity touches on the floor
What actually happens is that the entity is falling too fast, and so is blocked, just like in the problem above. Then the gravity is applied again, and it starts to fall again, but slower.
#3: You maybe managed to pass through the floor when falling from high enough.
If you’re falling really too fast (more than 8px/frame, the size of a block) and are a bit unlucky, the collision detection won’t “see” the floor, and your entity will clip through it.
#4: You maybe managed to get stuck inside, or jump through a block.
Since we’re independently checking the horizontal and vertical collisions, it’s possible to clip through a block when moving diagonally.


                  
                  Important
                  
                

The first three problems all have the same stem: AABB collisions don’t work well if you’re moving at more than 1px/frame. The fourth issue can be avoided with a specific diagonal check.


So, aside from bandaids like a careful level design and a velocity cap, how to solve this properly?
Collision Response
Like an annoying Souls-like boss, you realize with horror that collision detection was actually the easy first phase. Now comes the dreaded collision response. Ok maybe not dreaded, but we have to think a bit harder.
So far, with our naive implementation, we mostly avoided the response. We detect if an entity will collide with a wall, and we just stop the entity to prevent the collision from happening. It’s a response, but clearly not the best.
An easy solution is to move the entity against the wall before stopping it, instead of just stopping it before it collides.
The updated code is available here.

What did we change?
First, we make sure to cap the velocity to 7px/frame when falling. That should prevent most clipping issues. Then, if we detect a collision when moving, we “teleport” the entity right against the block it’s about to collide with.
-- collision left
if self.dx &lt; 0 and (isColliding(self.x + self.dx, self.y)
  or isColliding(self.x + self.dx, self.y + 7)) then
  -- move the player against the left wall,
  -- if our x coord isn&#039;t already a multiple of 8
  if self.x % 8 ~= 8 then
    self.x = flr(self.x / 8) * 8
  end
  self.dx = 0
end
We also added a simple diagonal check to avoid this particular edge case.
So while it doesn’t solve all problems and can be a bit limited, this collision code is now an adequate starting point for most games.


                  
                  Further improvements 
                  
                

For the sake of simplicity, in these examples, everything has a size of 8 pixels. The logic will still be the same if your entities have a different width and height, and you’ll just have to adapt the values.

 ]]></description>
    <pubDate>Mon, 07 Apr 2025 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>On Liminal Spaces</title>
    <link>https://scambier.xyz/blog/on-liminal-spaces</link>
    <guid>https://scambier.xyz/blog/on-liminal-spaces</guid>
    <description><![CDATA[ I had a student job nearly twenty years ago, in a huge library. One of my tasks was to retrieve stuff in the underground archives.
~
Now imagine. You’re in the elevator, going down to -5, and the doors open on this massive floor that someone only briefly showed you a few days ago. The fluorescent lamps of the central aisle automatically light up as you step out and begin walking. On your left and right are dozens of narrow rows of shelves, that are so long that the light can’t reach the end. And you’re there all alone, walking past these identical never-ending shelves that end in darkness, looking for some decades-old newspaper. The further you’re going from the elevator, the more uneasy you are. You hope that the motion detectors all work so the lights don’t suddenly go out.
But you’re on the wrong floor, what you’re looking for is actually in -6. So you take the service stairs down, and arrive on this floor that somehow looks even more oppressive. It’s smaller, the ceiling is lower, the shelves are packed tighter, and at the end of the floor, in a corner, there is this massive metal door. It looks so out of place, what could be on the other side? But you don’t dare open it, not a chance. You know ghosts and monsters don’t exist, you’re an adult, but you won’t open this door.
A few days later, when you go down again, you’re with another student. He also noticed this door, and also did not open it. Now, since you’re no longer alone, why not open it, just to peek? You unlock the heavy latch and pull the door. There is a sudden draft with a howl, the wind almost closes the door by itself, as if you’re the first ones to open it in years.
And inside, there’s literally nothing.
An empty dark space of naked concrete floors and walls, that is only barely lit up by the light coming from the open door. Not a single piece of furniture, a light fixture, a cable, or a pipe. Just an empty room, too dark to see through, sealed with a heavy metal door. You close it, disappointed but not really relieved. You ask one of the librarians what’s inside, what’s the purpose of that door, and they don’t know. It was probably a maintenance storage. Maybe. You’ll never get an answer.
~
And twenty years later, I still think about those shelves and that door whenever I see something about liminal spaces.
This was originally posted as a comment on Tildes, in a discussion about liminal spaces.



function blink() {
	$(&#039;#lamp&#039;)
		.fadeTo(50, 0.6)
		.fadeTo(50, 0.3)
		.fadeTo(50, 0.8)
		.fadeTo(50, 0.6)
		.fadeTo(50, 0.7)
		.fadeTo(50, 0.5)
		
		.fadeTo(50, 0.6)
		.fadeTo(50, 0.3)
		.fadeTo(50, 0.8)
		.fadeTo(50, 0.6)
		.fadeTo(50, 0.7)
		.fadeTo(50, 0.5)
		
		.fadeTo(50, 0.5)
		.fadeTo(50, 0.2)
		.fadeTo(50, 0.7)
		.fadeTo(50, 0.5)
		.fadeTo(50, 0.6)
		.fadeTo(50, 0.4)
		
		.fadeTo(50, 1.0)
		.fadeTo(200, 1.0)
		
		.fadeTo(50, 0.6)
		.fadeTo(50, 0.3)
		.fadeTo(50, 0.8)
		.fadeTo(50, 0.6)
		.fadeTo(50, 0.7)
		.fadeTo(50, 0.5)
		
		.fadeTo(50, 1.0)
}
setTimeout(blink, 18_000)

 ]]></description>
    <pubDate>Wed, 15 Jan 2025 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>2024 Recap</title>
    <link>https://scambier.xyz/blog/2024-recap</link>
    <guid>https://scambier.xyz/blog/2024-recap</guid>
    <description><![CDATA[ So, what did I do this year?
Code and Tech
Svelte 👎
I discovered Svelte (back then in version 3) when I started working on Omnisearch. I liked it because its syntax was truly magic, even though it had some glaring issues that would probably be fixed in later versions.
This year, I used Svelte 5 to write SCRABEUL, a small daily word game, and I must say I’m not impressed. They totally changed their philosophy — with reason —, but doing so they changed their framework into what I feel is a subpar Vue copycat. It also doesn’t help that I disagree with several design decisions taken by SvelteKit.
I’m honestly disappointed they didn’t try harder to iterate on their original philosophy, because now they’re just another JS framework with almost nothing to set it apart.
PocketBase 👍
I needed a backend solution to store SCRABEUL leaderboards, and I found PocketBase to be an excellent solution. It’s a self-hosted Database-As-A-Service that gives you authentication, migrations, a customizable REST API, hooks, …
It’s meant to be used and called “as-is” by the client app, and not hidden behind your own backend, though the latter is obviously possible (just not recommended). The only downside is that while you can totally write your code in JavaScript, it’s executed by an embedded engine and comes with pretty strong caveats.
Copilot 👎
After using it for nearly two years, I finally cancelled my subscription. A small explanation here.
Rust 😶
Not much code written in Rust this year, unfortunately. I just need a good idea to take advantage of Tauri. In the meantime, I’ll try to find small enough projects that I can whip up in a weekend or two.
Gaming
Dwarves ⛏
According to Steam, my most-played genre this year is… “Dwarf”. I actually played a lot of Deep Rock Galactic and its Survivor spin-off. That, and Warhammer Vermintide II. Coop FPSes are the perfect genre for gamer dads.
Automation 🤖
My love story with Shapez was short and intense. I played for 30h in something like 4 days, and then uninstalled it before getting more addicted. The sequel is in my wishlist, but I’ll wait for it to get out of early access.
Satisfactory was not satisfying at all. I want to automate stuff, not walking back and forth to craft a goddamn Minecraft mechanical mansion.
I’ll also throw Oxygen Not Included  in the “automation” category. I’d like to like it more, but each time I play it, I spend a whole afternoon without accomplishing anything. I have fun, but I feel like I do not progress at all. And then I forget the game for a few months, start a new colony, rinse and repeat.
Roguelikes and Roguelites 🎲
Only sure values here.
Caves of Qud is a delight, and I should play it more. It also runs and plays great on the Steam Deck but the sprites are wayyy too tiny. If you’re looking for a lighter traditional roguelike that is made to be played with a pad, Jupiter Hell is right here. It’s literally a turn-based Doom.
Dead Cells and The Binding of Isaac with their respective DLCs are probably the two games I’ll never uninstall. I also certainly won’t ever finish them.
Balatro came and went. Big numbers go brrr for the first 20 hours, and then you understand that playing pairs is the safest winning strategy.
Nova Drift is an interesting shoot-em-up with a large variety of possible builds. Not too dissimilar of ARPGs, actually. Speaking of which…
Action-RPGs ⚔
Just like Diablo 3 was my comfort game on the Switch, Diablo IV is my comfort game on the Steam Deck. It’s dumb fun, fluid, pretty, and there’s always something to do to get bigger numbers. I don’t need to think, I just smash demons. Now if only it could support being put into sleep mode without disconnecting or crashing, that would be perfect. At least I can do that with D3 on the Switch 🙄
So as a slower-paced and offline alternative, I just bought Last Epoch. The build and crafting systems are nice to play with and allow for a ton of customization, but the game definitely needed more time in the oven before reaching 1.0. But at least it’s not a blatant “game-as-a-service”.
I tried to like Grim Dawn. I really tried. But as rich as it is, it just feels antiquated, and is a pain to play on the Deck.
Others
I still have to finish Strange Horticulture, Forza Horizon 4, and Batman Arkham Knight.
Tiny Glade is a pretty toy in the same vein of Townscaper. I got 6 relaxing hours out of it, and will probably relaunch it in a few months when it’s a bit more fleshed out.
Not on screens
Clay Sculpting! ]]></description>
    <pubDate>Fri, 27 Dec 2024 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>RoguelikeDev Does The Complete Roguelike Tutorial - 2024</title>
    <link>https://scambier.xyz/blog/roguelikedev-does-the-complete-roguelike-tutorial---2024</link>
    <guid>https://scambier.xyz/blog/roguelikedev-does-the-complete-roguelike-tutorial---2024</guid>
    <description><![CDATA[ 

                  
                  Where&#039;s the code? 
                  
                

Sorry, no public repository yet! The engine code is a WIP and isn’t ready for distribution, and the game repository has at least one copyrighted asset. I’ll share the relevant snippets for now.


Like every year, the subreddit /r/roguelikedev is organizing its summer tutorial series. The goal is to follow a Python-based roguelike tutorial in 8 parts — one per week —  and have a playable game at the end of it. Obviously you can use whatever tech stack you’d like and not follow the tutorial at all. It kind of acts as a gamejam with mutual emulation and weekly reports from participants. For many, it’s a great summer side project. The announcement post is here.
I’ll try to keep up, but it’s the third of fourth time I’m doing it, and each time I either lose interest or fall of the bandwagon because I’m straying too far from the source material and end up with 0 ideas to wrap up in time… Anyway.
Week 0 - Preparation
Since my favorite hobby is to make half baked engines instead of games, I actually already have everything on hand. I’ll make the game in TypeScript, with a &lt;canvas&gt; element. No rust this time, I want something as frictionless as possible, to focus only on the code (and not on some build issue). I also want to iterate quickly.
So I have this pico8-slash-love2d-inspired library that’s absolutely ready for something as graphically simple as a roguelike.
Week 1 - Draw the @ and move it around
Reddit post
So far so good.
I bought the tileset “Oh no, more goblins!” a few weeks ago, for an unrelated prototype, so I’m going for something prettier than a simple @. The CRT shader I took from here helps a lot, too.

Look at them little arms full of chromatic aberrations 📺
I’ve already over-engineered some things, but I’ll probably rewrite the actor/entities/composition code a few times before settling on something that kinda works.
export class Position extends Component {
  constructor(
    public x: number,
    public y: number
  ) {
    super()
  }
}
 
export class Sprite extends Component {
  constructor(
    public spritePos: [number, number],
    public color: Color
  ) {
    super()
  }
 
  public draw() {
    const { x, y } = this.actor.getComponent(Position)
    E.draw.sprite(this.spritePos, x * 12, y * 12, E.FLIP.Horizontal)
  }
}
 
class Hero extends Actor {
  constructor() {
    super()
    this
      .addComponent(new Position(1, 1))
      .addComponent(new Sprite([15, 8], Color.Green)
    )
  }
}
A full ECS is probably overkill, and I don’t want to use inheritance for actors, so I’ll go with a more classical composition pattern, where each component is responsible to update() and draw() itself.


                  
                  Web build - Week 1 
                  
                

There’s not much to do yet, but you can try the “game” here.


Week 2 - The dungeon
First, a bit of refactoring
Actually, that Actor thing sucked, so I’ve already refactored that part to use my in-house ECS library.
// Component factories
export const CmpSprite = Component&lt;{ sprite: [number, number]; color: Color }&gt;()
export const Position = Component&lt;{ x: number; y: number }&gt;()
export const IsPlayer = Component()
 
// Create a player entity
export function makePlayer(): Entity {
  return world.spawn(
    CmpSprite({ sprite: Sprite.Goblin, color: Color.White }),
    Position({ x: 1, y: 1 }),
    IsPlayer()
  )
}
 
// Render drawable entities
export function sysDrawSprites(world: World) {
  for (const [_e, sprite, pos] of world.query([CmpSprite, Position])) {
    E.draw.setColor(Color.White, sprite.color)
    E.draw.sprite(sprite.sprite, pos.x * 12, pos.y * 12, E.FLIP.Horizontal)
  }
  E.draw.resetPalette()
}
Level generation

The algorithm isn’t too complicated, but gives interesting results:

Fill the map with wall tiles, and randomly place a few non-overlapping rooms.
Create a weighted graph of our map with the following rules (adapt the values as needed):

Dungeon outer walls are impassible
Rooms walls are 100_000
Floor tiles are 1_000
Other walls are a random pick of [1, 1, 100, 100_000]


Use a pathfinding algorithm (in this case, A-star) to find a corridor’s path from the center of a room to the next
Dig a tile out of this corridor
Go to 2

The weightings used to make the graph force the corridors to only dig into rooms when necessary (they prefer to go around them), but merge with existing paths when possible. The random values make them a bit wobbly for a more organic look. To make the rooms less blocky, I also fill a few corners at random without blocking the path. This algorithm has a tendency to create dead-ends, but it should be easily mitigated by adding a straight path between one or two pairs of close rooms.
I also started implementing doors, but they don’t open yet ¯\(ツ)/¯


                  
                  Web build - Week  2 
                  
                

This week’s version is playable here. Use F5 to get a new dungeon, and F6 to toggle the CRT shader.


Week 3 - Field of view (FoV) and enemies
For the Field of View (and Fog of War), I’m quite fond of the “symmetric shadowcasting” algorithm, as I’ve written on it before, and the original python code is relatively easy to translate into any other language.

And for the enemies, Instead of goblins and trolls (our hero is already a goblin), let’s spawn bats and snakes.
export function makeSnake(x: number, y: number): Entity {
  return world.spawn(
    Name({ name: &#039;Snake&#039; }),
    IsEnemy(),
    IsBlockingTile(),
    CmpSprite({ sprite: Sprite.Snake, color: Color.Green }),
    Position({ x, y }),
  )
}

Tile bitmasking
I feel like I’m becoming an expert on tile bitmasking. Each time I implement it (and I think it’s the 4th time, at least), I refine my existing code.
Depending on the surrounding cardinal (N S W E) and diagonal (NW NE SW SE) neighbors, there are 47 possible relevant combinations for every tile in your dungeon. So if you want to be thorough, you need 47 different sprites to represent all those possibilities. But if you don’t, having 16 different sprites is just enough. And conveniently enough, you can use CP437 characters to represent them all.
// Spritesheet references
const walls = {
  &#039;╣&#039;: [0, 20],
  &#039;║&#039;: [1, 20],
  &#039;╗&#039;: [2, 20],
  &#039;╝&#039;: [3, 20],
  &#039;╚&#039;: [4, 20],
  &#039;╔&#039;: [5, 20],
  &#039;╩&#039;: [6, 20],
  &#039;╦&#039;: [7, 20],
  &#039;╠&#039;: [8, 20],
  &#039;═&#039;: [9, 20],
  &#039;╬&#039;: [10, 20],
  &#039;╥&#039;: [11, 20],
  &#039;╨&#039;: [12, 20],
  &#039;╞&#039;: [13, 20],
  &#039;╡&#039;: [14, 20],
  &#039;█&#039;: [15, 20],
} as const
And then you can map them to the 47 combinations, with duplicates where applicable.
    const masks: Record&lt;number, Readonly&lt;[number, number]&gt;&gt; = {
      0b01111111: walls[&#039;╔&#039;],
      0b00111101: walls[&#039;═&#039;],
      0b10111111: walls[&#039;╗&#039;],
      0b10001010: walls[&#039;╔&#039;],
      0b01000110: walls[&#039;╗&#039;],
      0b00001010: walls[&#039;╔&#039;],
      // etc.
A bit tedious, but 100% worth it.
Bumping into enemies
First, a system to keep an up-to-date list of who is where:
export function sysRefreshMapOccupations(world: World) {
  map.emptyOccupiedTiles()
  for (const [e, pos] of world.query([Position])) {
    map.occupyTile(pos.x, pos.y, e)
  }
}
And then it was just a matter of checking the destination before moving. I added this small method on my map instance:
  public canbeBumped(x: number, y: number): boolean {
    // Not a wall, but something else that can be interacted with (a door or another entity)
    return !this.isWall(x, y) &amp;&amp; (this.occupiedTiles[this.xy_idx(x, y)].length &gt; 0 || this.isDoor(x, y))
  }
🤔 I’m still debating if I should use entities for doors, instead of treating them as special dungeon tiles…
So, what happens when bumping into something? I’m using a small PubSub object to dispatch an event: eventBus.publish(Event.Bump, { pos }). Usually I’m not a fan of this pattern, as it can make it difficult to follow the logic, but the alternatives were spaghetti code that does everything, or “event components” passed through the ECS which would add another layer of indirection.


                  
                  Web build - Week  3 
                  
                

This week’s version is playable here. You can now open doors and bump into enemies.


Week 4 - Fighting and UI

 Implement a camera
 Add a message log
 Add a HP bar
 Add a minimap
 Display information when hovering an entity with the mouse
 Rework the RNG module from my engine
 Copy the combat logic from Brogue
 Semi-random movement for the bats
 The snakes follow
 Take damage
 Proper death (= the game doesn’t crash when the player dies)

Writing hard, checklists easy.

I’ll try to complete the tutorial, but will certainly stop the official schedule of 2 parts per week here. This is taking too much of my time and I have other priorities ✌


                  
                  Web build - Week 4 
                  
                

This week’s version is playable here

 ]]></description>
    <pubDate>Fri, 12 Jul 2024 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>Markdown Sucks</title>
    <link>https://scambier.xyz/blog/markdown-sucks</link>
    <guid>https://scambier.xyz/blog/markdown-sucks</guid>
    <description><![CDATA[ I don’t understand how Markdown won the battle of “simple markup” languages. Out of the three big nerd syntaxes (Markdown, Org, AsciiDoc), it’s the worst.



































LanguageMy momcan use itRich setof featuresWell-definedstandardMS Docs✅✅✅AsciiDoc❌✅✅Org❌✅✅Markdown❌❌❌
In a parallel universe, Obsidian uses AsciiDoc, and Markdown is remembered as some half-assed idea that never got any traction.
(Org-Mode users are still doing their thing though, they’re probably a constant in all parallel universes.)
“But Simon! Markdown is popular because it’s simple and straight to the point!”
Well, Obsidian having non-standard features like transclusions, admonitions, or even tags, plus the 1 million+ installations of plugins such as Dataview, Tasks, or Templater tend to disagree with you.
People want more complex features. Also, “simple” doesn’t mean “dumb”. Markdown is dumb.
And yet, I’m glad it exists.
Mainstream apps like Discord, Teams, Slack, Notion, etc. incorporate its syntax in text fields. Editors like Obsidian, Bear, Joplin, and many others are bringing it to non-tech people, who discover that rich documents can be saved as plain text, the most long-lasting format ever.
It could have been better, but Markdown is bringing (back) plain text to the masses, so that’s cool. ]]></description>
    <pubDate>Thu, 06 Jun 2024 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>Advent of Code 2023</title>
    <link>https://scambier.xyz/blog/advent-of-code-2023</link>
    <guid>https://scambier.xyz/blog/advent-of-code-2023</guid>
    <description><![CDATA[ Advent Of Code is a series of 25 2-part puzzles that must be solved programmatically. The answer of all puzzles is always an integer value.
I’ll try to make a small visualization for each exercise, so I’ll use PICO-8 (Lua) or TIC-80 (TypeScript), since both engines make it possible to quickly draw something with just a few more lines.
Day 1


Exercise: adventofcode.com/2023/day/1
Solution: git.scambier.xyz/scambier/adventofcode2023/src/branch/master/day1.p8
Language used: PICO8’s Lua 1

This first day was (surprisingly?) quite easy. Looking at posts on Mastodon, it seems that many participants struggled with regexes and edge cases like oneight and twone. Since I don’t have access to regexes with PICO-8, I just checked for substrings 🤷‍♂️
And since PICO-8 can’t easily handle big numbers (signed integers can’t go over 2^15-1), I found a function that adds strings.
I wasn’t really inspired for this 1st day visualization, so I lazily printed out the values as the algo was processing them.
Day 2


Exercise: adventofcode.com/2023/day/2
Solution: git.scambier.xyz/scambier/adventofcode2023/src/branch/master/day2.p8
Language used: PICO8’s Lua

I had to cheat a bit today. Since PICO-8’s Lua is so limited in features, splitting and parsing strings can get tedious very fast; so I “manually” cleaned some spaces from the input, to make it easier to split lines.
The problem was overall simpler than yesterday’s, just a bunch of ifs and additions. Almost half of my code is dedicated to the falling snowflakes. This is a classic animation: snowflakes fall 1pixel/second, I used a sin() function to make them sway left and right. They stop falling once they reach the bottom of the screen, or if there’s already a snowflake right beneath them. Each snowflake corresponds to a red/green/blue cube from the input.
Day 3

Yes I know that’s not how gears work, thank you very much.

Exercise: adventofcode.com/2023/day/3
Solution: git.scambier.xyz/scambier/adventofcode2023/src/branch/master/day3/main.ts
Language used: TypeScript

This one was hard.
Not that the problem is particularly hard — finding adjacent items on a 2D grid isn’t really complicated — but I had a bug in my first Lua script, and I absolutely could not find it. The example input was correct, the logs were correct, the different test cases were correct, and yet the end result was incorrect. I trashed the code and tried a different algorithm in TypeScript, which worked. I still don’t know what was wrong in my first draft 😑 I can only guess it was some edge case that I overlooked…
Anyway, once the 1st part was finally done, the second part went smoothly.
I also wrote a small animation that fit the “gears” theme ⚙️ I take 1 out of 10 so-called gears from part 2, report their position on the TIC-80 space, print a * surrounded by the corresponding numbers, and make those numbers rotate around the cog.
Day 4


Exercise: adventofcode.com/2023/day/4
Solution: git.scambier.xyz/scambier/adventofcode2023/src/branch/master/day4.p8
Language used: PICO-8’s Lua

I had to cheat again by pre-cleaning the input. I’m ok with this as it’s kinda necessary with PICO-8 if you want to spend more time solving the problem than cleaning the input.
Anyway, today was quick and easy, perfect for a Monday. I had to read the second part a few times to make sure I understood it correctly. No wonder those elves have issues with that many scratch cards. Gambling addicts.
I was not really inspired for the visualization, as PICO-8 is difficult to work with big numbers. I mean, I do have ideas, but they’re kinda hard to combine with the input/output numbers from the exercises without spending hours on the execution.
Footnotes


This a subset-slash-custom version of Lua, with most of the standard lib removed. ↩


 ]]></description>
    <pubDate>Fri, 01 Dec 2023 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>The Ultimate Beginner Guide for Obsidian</title>
    <link>https://scambier.xyz/blog/the-ultimate-beginner-guide-for-obsidian</link>
    <guid>https://scambier.xyz/blog/the-ultimate-beginner-guide-for-obsidian</guid>
    <description><![CDATA[ If you’re curious to try Obsidian — the note-taking app — or even better yet, if you’ve just started and feel overwhelmed, I’ve spent the last two years carefully curating 54 plugins that will help you to get started as easily as possible.
Nah I’m kidding.

You want to know how to use Obsidian efficiently?

Open it
Write something
(optional) Make links and/or add tags

Wanna be productive and get things done? Watching 3 hours of videos to understand zettelkasten or johnny decimal won’t help. Not everyone needs to take notes on every movie they watch, every book they read, or every podcast they listen to. You probably don’t need to carefully record the minutes of all your daily standups. Obsidian, LogSeq, or Joplin won’t make you a superhero, and you won’t fail in life1 if you only write down the few important things that you need to remember.
Maintaining a second brain is supposed to free your first one, not make it even busier and more stressed.

1. You can definitely fail in life, but your lack of notes is unlikely to be the main cause. ]]></description>
    <pubDate>Mon, 20 Nov 2023 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>Why I like using AI programming tools</title>
    <link>https://scambier.xyz/blog/why-i-like-using-ai-programming-tools</link>
    <guid>https://scambier.xyz/blog/why-i-like-using-ai-programming-tools</guid>
    <description><![CDATA[ _My views have evolved since I first wrote this post in 2023. _
Original post
A few days ago, I stumbled on this blog post, titled “Why I will likely never use AI programming tools”.

But the act of thinking and figuring out a problem, solving it, writing some code, fixing bugs…
Especially creating complex systems that interact with each other, with a lot of moving parts! (Hi, gamedev)
It’s fun to me.

This is exactly why I like using AI programming tools. Same reasoning, opposite conclusion.
AI tools allow me to completely evacuate the boring parts of programming: the boilerplate, the dumb loops, the language idiosyncrasies. When you consider AI tools - like the aptly named Copilot - as “helpers” that generate pseudo-code, you can focus on the fun parts: the problem solving, the architecture, the design.
The AI doesn’t write code for you, it writes code with you.
I feel like ranting against the AI is like ranting against the invention of the compiler. Yes you can find joy in writing Assembly code, but… well.
Now maybe that will change in 6 months, or a year or two. Maybe the AI will be so good that it will write code for you. Maybe I’ll be out of a job before 2030, but if that happens, I’m sure going against the flow won’t help. For better or worse.

2024-06-04 Update
From a Bluesky user:

Google explaining the “glue pizza” search result by stating EXPLICITLY that the wrong answer was given BECAUSE nobody had ever asked that question in the training data tells you that what we have isn’t artificial intelligence. It’s a server that copies, changes slightly, and pastes. Not intelligence

Awful for factual answers, perfect for programming : AI is a giant copy-paste machine, so this is exactly what you need for code 90% of time. Even if your program is groundbreaking and 100% original, most of its snippets are not.

Source
Glue pizza reference: Google Is Paying Reddit $60 Million for Fucksmith to Tell Its Users to Eat Glue


2024-12-24 Update
I cancelled my Copilot subscription.
It’s equally useful for some things (boilerplate code, unit tests, api discovery in an unfamiliar language or framework), and terrible for everything else. So in the end, I don’t think it saves me any time. What’s worse is that the most interesting thing it offers — code/api discovery — makes me lazy and not willing to understand what “I” write. I consider using Copilot for these tasks to be a net loss.
Many people have written much more articulate thoughts on it, so I’ll left it here. I’m glad I gave it a thorough try, though.

2025-08-06 Update

Status at home: no AI auto-completion, though I still use Copilot free tier as a (wasteful) rubber duck. I don’t miss it when I’m out of tokens, though.
Status at work: Copilot is also useful as a rubber duck for large, complex projects. Code completion is still a mixed bag, and I still estimate the net gain as zero.
 ]]></description>
    <pubDate>Wed, 21 Jun 2023 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>Improving Symmetric Shadowcasting for Tile Bitmasking</title>
    <link>https://scambier.xyz/blog/improving-symmetric-shadowcasting-for-tile-bitmasking</link>
    <guid>https://scambier.xyz/blog/improving-symmetric-shadowcasting-for-tile-bitmasking</guid>
    <description><![CDATA[ (That title definitely makes me look smarter than I am)

On my spare time, I like to write code for my forever-work-in-progress traditional roguelike. I don’t expect to finish it, since it’s mostly a training ground for my favorite language/engine of the day, but I still enjoy trying new things. At this moment, I’m using Rust, and was motivated by the Porklike tutorial to implement drawn tiles with bitmasking. The general idea is to have nice-looking walls that form a cohesive structure, instead of a repetitive pattern of the same tile.
A point to consider, though, is that when used naively, a bitmask can reveal information to the player. As an example, this cropped screenshot of Caves of Qud:

The player can’t see behind those two wall tiles. We have the same amount of information for both tiles, yet one is clearly the beginning of a longer wall, while the other is just a pillar. I’m not saying this is a bad thing; maybe it doesn’t matter, maybe it’s even desirable to have this information. Personally I find it removes a bit of exploration and surprise.
A simple solution to this, and the one I chose, is to only apply the bitmasking algorithm on revealed tiles, and assume that non-revealed tiles could be walls; this way, a wall is not revealed to be a pillar until you’ve seen all the floor tiles around it. The small counterpart of this method is that some sprites will suddenly switch (e.g. from “wall part” to “pillar”) according the the tiles you’ve seen or not.
The algorithm I use to calculate the field of view (FoV) is the popular “Symmetric Shadowcasting”. It works really great, but there is a small problem:

This is what the algorithm sees:

On my screenshot, the green tiles show completed walls because they’re surrounded by floor tiles, but the ones in red are incomplete walls, because we technically don’t know what is under the ?.
So, since it’s not a wall, it’s obviously a floor. Is it a bug in the algorithm? No, not really.
A floor tile is considered to be inside the FoV when you can see its center point. This is the symmetric part of the algorithm, whose goal is to make sure that if you can’t see an enemy, then the enemy can’t see you either.
The missing floor happens to be technically right outside the FoV. That’s why the algorithm doesn’t reveal it, so as not to also reveal a potential enemy who couldn’t see you. While it is a desirable property for gameplay reasons, visually it just looks like a random hole in the line of sight.
Let’s adjust the algorithm, and change the scan() function from this:
### fov_compute.py
 
def reveal(tile):
	x, y = quadrant.transform(tile)
	mark_visible(x, y)
 
def scan(row):
	prev_tile = None
	for tile in row.tiles():
 
		# The tile is revealed if it&#039;s a wall,
		# or if it&#039;s a floor that satisfies the symmetry rule
		if is_wall(tile) or is_symmetric(row, tile):
			reveal(tile)
 
# (more code)
To this:
### fov_compute.py
 
-def reveal(tile):
+def reveal(tile, partial):
	x, y = quadrant.transform(tile)
-	mark_visible(x, y)
+	mark_visible(x, y, partial)
 
def scan(row):
	prev_tile = None
	for tile in row.tiles():
 
		# The tile is revealed if it&#039;s a wall,
		# or if it&#039;s a floor that satisfies the symmetry rule
-		if is_wall(tile) or is_symmetric(row, tile):
-			reveal(tile)
+		reveal(tile, is_floor(tile) and not is_symmetric(row, tile))
 
# (more code)
Now the mark_visible() callback function will be called with an additional parameter that specifies if the tile is an non-symmetrical positioned floor. You just need to manage the partial case in your callback, and flag such tiles as “visited”, but not “in view”.
The result:


The tile is correctly registered as a floor, it’s displayed as such, and if there’s an enemy on it, they will stay hidden until we’re coming close enough. ]]></description>
    <pubDate>Wed, 23 Nov 2022 00:00:00 GMT</pubDate>
  </item>
<item>
    <title>Indexing and Searching Four Thousands PDFs</title>
    <link>https://scambier.xyz/blog/indexing-and-searching-four-thousands-pdfs</link>
    <guid>https://scambier.xyz/blog/indexing-and-searching-four-thousands-pdfs</guid>
    <description><![CDATA[ 
Ok, maybe not exactly four thousand, but close enough.
During my free time, I’m developing Omnisearch. Behind that so original name lies a search plugin for the note-taking app Obsidian. In short, Omnisearch uses an industry standard algorithm to return “smart” results: when you type in a query, each note is given a weight, and the notes with the higher weights are sorted on top.
One of the most important features I wanted to add was PDF indexing. Now, I know that PDFs are a can of worms, so I wasn’t going to start working on it without making sure that a) I could do it and b) it wouldn’t eat my evenings for 6 months.
PDF.js, round one
Since Obsidian is an Electron app, PDF.js1 was the obvious choice - even more obvious that I quickly found out Obsidian has it bundled. I think it took me less than an hour to wrap the feature, and ship a new version of Omnisearch. That was easy, I was wondering why it wasn’t even a default feature.
Except that if you have more than two dozens PDFs, Obsidian crashes. A hard, unrecoverable, Electron crash. Whoops, let’s rollback and try again.
PDF.js, round two
So, I tried again, but this time I added PDF.js as an Omnisearch dependency. I wasn’t too happy to do it because it made the main.js file jump from 70KB to 1.34MB. See, PDF.js is great, but it’s not really an embeddable library that plays well with tree-shaking. It’s developed by Mozilla as Firefox’s PDF viewer, so not really something you’d use to just parse PDFs. But, hey, it works. it added some freezes, but that was an encouraging start. Until crashes appeared again.
It didn’t matter if I was parsing them fast or slow, the end result was a crash. The more files, the more likely Obsidian was going to crash. And during all those tests, there was a small issue that was bugging me, and that I wasn’t able to fix. Each time PDF.js was starting, there was this message in the console:

Warning: Setting up fake worker 2

I tried everything, I never managed to make it go away. Is it related to the crashes? Maybe. Anyway, back to square one.
Let’s sprinkle a bit of Rust (and change the bundler)
Luckily I have some experience in the language Rust, and while I hadn’t done it yet, I knew it’s a language that you can (theoretically) easily compile to WebAssembly (wasm). Wasm is a way to take a fast, lower-level language, and make it run in a webapp, alongside the easier - but relatively slower - JavaScript.
Thanks to an excellent example project3, I quickly made a prototype with a fork of pdf-extract4, though it required me to change my bundler from esbuild to Rollup. I did try to make wasm work with esbuild, but it was seamless with Rollup, so I didn’t push it.
Extracting PDFs’ texts with wasm was working, but it was much slower than I expected, and worse, Obsidian was completely freezing during indexing. Indeed, wasm execution was taking 100% of the main thread, which is usually not good for the user experience.
Use ALL the cores!
Now that I’ve added wasm, let’s delegate the heavy work to web workers. Those little fellas are a way to use multi-threading in JavaScript. I mean real multi-threading, not Promises. Give them a long-running, UI-blocking task - like, say, reading PDFs - and they’ll work in the background, then send you the result once they’re done.
Phew. Does it work now? Can we index PDFs without blocking everything? Yes! Oh wait, no.



I mean, yeah, 5000 PDFs. I wasn’t really surprised that Obsidian couldn’t keep up, but I had to mitigate it. A plugin causing a hard crash is inexcusable, so it had to be fixed before going out of beta.
Honestly at that point, I was getting kinda exhausted. What was supposed to be a one-hour job had already become a 15-evenings slog. It was working on my machine (as always) with a dozen PDFs, but the more files, the more Obsidian was struggling, before completely giving up. And the crashes were not unlike the ones I experienced with PDF.js… It took me two days (and a lot of sloooowwwww reloads) to find the actual issue.
/* abridged code */
class PDFManager {  
  public async getPdfText(file: TFile) {  
    const data = new Uint8Array(await app.vault.readBinary(file)
    const worker = new PDFWorker({ name: &#039;PDF Text Extractor&#039; })  
    worker.postMessage({ data, name: file.basename })
    worker.onmessage = (evt: any) =&gt; {
	    /* index the text */
	}
See what’s wrong? I was spawning a new worker for every PDF, and Electron (the engine under Obsidian) doesn’t like that. Obvious in hindsight, but hard to see when getting tired of working every evening on a feature you’re not really going to use yourself.
Those workers are never cleaned, they hang out in memory (even when doing nothing) and inevitably end up causing slowdowns and crashes.
So I promptly fixed that with a pool of workers, and ta-da!

All my CPU cores happily extracting text from PDFs, in parallel, while Obsidian stays mostly unaffected. Mostly? Yep, sometimes it lags. Or freezes. For a few seconds only. Ok, maybe twenty. Sigh 😫
The never-ending optimizations
This post is already long enough, and I’m not a good writer. Sequence of tenses is already hard enough in French… Anyway.
Let’s say that, today, October 25, exactly one month after I started working on this feature, it finally appears to be silky smooth. It’s not super fast, the initial cache building takes time, but it’s smooth. I had to completely refactor Omnisearch, make small optimizations that definitely mattered at scale, and consider uses of Obsidian that are radically different than mine.
I also had to throw away the old cache implementation, after struggling with it for three days. Because I had to deliver the PDF baby, and the cache was in the way. PDFs themselves are still cached, but for now, the search index is still rebuilt at every startup. A bit inconvenient for some users, but I’ll come back to it in a few days, once I’ve spent a few evenings playing video games instead of programming.
All for this

A single opt-in checkbox.
And I’m really, really proud of it. Not only did I learn a lot while working on this, but there’s also nothing better than a complex piece of work being completely invisible to the end-user.
I’d also like to thank tekwizz123 and Benny, who provided incredibly useful feedback during this month of development. Four thousand thanks to both of you, and of course, to the users who take time to fill Github issues.
If you’re interested to see how the feature developped, day by day, you can read the Github issue thread5
Footnotes


mozilla.github.io/pdf.js/ ↩


www.google.com/search%22warning%3A+Setting+up+fake+worker%22 ↩


github.com/trashhalo/obsidian-rust-plugin ↩


github.com/jrmuizel/pdf-extract ↩


github.com/scambier/obsidian-omnisearch/issues/58 ↩


 ]]></description>
    <pubDate>Tue, 25 Oct 2022 00:00:00 GMT</pubDate>
  </item>
  </channel>
</rss>