Josh On Designhttp://joshondesign.com/Art, Design, and Usability for EngineersTue, 19 Mar 2024 10:51:50 +0000Circuit Python Watch Statushttp://joshondesign.com/2023/07/25/circuitpython-watchhttp://joshondesign.com/2023/07/25/circuitpython-watchTue, 25 Jul 2023 19:22:58 +0000<p>It&#x27;s been a bit since I&#x27;ve posted on my round screen watch project. Most of my time has been taken up by work, travel, and family stuff, but I did have a few seconds to add a feature or two.</p><h3>New Screens</h3><p>The Waveshare board I&#x27;m using has a lipo battery charger inside it, but until now I hadn&#x27;t exposed it. There is now as screen that shows the current battery percentage, though I think my calculations might be off. <a href="https://github.com/joshmarinacci/waveshare_lcd_test/blob/5654c0a0956a5a00f0a4425b5c15463ee53bc4f2/watch/battery.py">What do you think</a>?</p><p>I also added a <a href="https://github.com/joshmarinacci/waveshare_lcd_test/blob/5654c0a0956a5a00f0a4425b5c15463ee53bc4f2/watch/timer.py">screen with a timer</a>. The actual timer itself works, but it won&#x27;t alert you when the timer ends if you switch to another screen. I need to figure out some sort of a background task.</p><p>I also added a moon phase indicator using some calculations I found on the web. Again I&#x27;m not sure it&#x27;s accurate, but the fun part is the star field particle effects behind the text. The hard part was figure out how to draw lots separate pixels (one for each star) without forcing the entire screen to refresh. The secret is to call bitmap.refresh() after each star is moved, rather than after drawing all the stars. If you do all the stars then you get one giant dirty rectangle that fills the whole screen and will be slow. If you call refresh after only changing two pixels (undrawing the old, then drawing the new), then in most cases you&#x27;ll have a dirty rect that is only 2 or 4 pixels in size, which is super fast to draw. Even with 50 particles, the total pixels drawn with my incremental approach is still a fraction of the full 240x240 screen size.</p><img src="https://joshondesign.com/images2/IMG_5878.jpeg" alt="z"/><h3>Sleeping</h3><p>The final big feature is making the watch sleep, though it&#x27;s only partially working. The screen will turn off after N seconds of no input. This more than doubles the watch lifetime on a single charge. However, the touch sensor and CPU are still running a full power, only the screen is off. The device does have a more advanced low power mode that will wake up the main CPU when a tap is detected, but unfortunately it requires monitoring a specific interrupt pin, which CircuitPython doesn&#x27;t support. There is some async io support in the form of a counter api, that I plan to investigate next.</p><h3>Pausing</h3><p>I&#x27;m happy with the progress of my watch, but I&#x27;m pausing this project for now while I work on a few other electronic projects. I recently built a set of animatronic eyes based on Will Cogley&#x27;s <a href="https://www.youtube.com/watch?v=Ftt9e8xnKE4">video tutoria</a>l, which I plan to expand into a singing tiki head. I&#x27;m also experimenting with the new <a href="https://docs.circuitpython.org/en/latest/shared-bindings/synthio/index.html">synthio CircuitPython API</a> to build a step seqencer in a gameboy-like form factor.</p><p>As always, the <a href="https://github.com/joshmarinacci/waveshare_lcd_test">full source is on github</a>.</p>CSS Text Style Builderhttp://joshondesign.com/2023/07/12/css_text_style_builderhttp://joshondesign.com/2023/07/12/css_text_style_builderWed, 12 Jul 2023 20:16:07 +0000<p>It’s a truism of the web that when something becomes free <em>it turns to crap</em>. This is largely due to advertising, tracking, and SEO hackers getting their crappy version of something to the top of the search index. When this has happened to a tool I want to use I can’t fix the underlying problem, but I can make my own non-tracking version and share it with the web, doing my little part to make the world a better place. Today it’s a CSS Text Style builder.</p><p>For my <a href="https://joshondesign.com/2023/06/08/pico_round_lcd">smart watch project</a> I want some cool looking fonts for the watch face. For the body text of the interface I can use nice fonts from <a href="https://fonts.google.com/">Google Fonts</a> and convert them to bitmaps (<a href="https://joshondesign.com/2023/06/25/cp_font_conversion">my tutorial here</a>). However, for the watch face itself, I want text with interesting full color effects like drop shadows, outlines, and 3D borders. Furthermore I don’t want an entire font, just the numbers and maybe a colon. Memory is precious.</p><p>At first I thought of writing drawing code to render these directly, but I realized I already have a great way to style text: CSS. The problem is the crappy SEOed tools I mentioned above. Being <em>me</em>, I solved it by building my tool. Behold: <a href="https://apps.josh.earth/texteffects/">CSS Text Effects</a></p><img src="https://joshondesign.com/images2/css-text-builder.png" alt="z"/><p>While still a pretty ugly UI, this tool does a couple of interesting things. The text style is actually a JS object with properties and constraints using <a href="https://zod.dev">Zod</a>, a Typescript library. Using these constraints the interactive sliders can be autogenerated from the spec. Furthermore the style is mirrored in the URL, so you can share a text style you make with anyone else by just <a href="https://apps.josh.earth/texteffects/?data=%7B%22sample%22%3A%22doop%22%2C%22fontSize%22%3A137%2C%22fontFamily%22%3A%22Josefin%20Sans%22%2C%22fontWeight%22%3A%22600%22%2C%22color%22%3A%7B%22h%22%3A138.59304028888081%2C%22s%22%3A0%2C%22l%22%3A0%2C%22a%22%3A1%7D%2C%22backgroundColor%22%3A%7B%22h%22%3A352.71844660194176%2C%22s%22%3A0.980952380952381%2C%22l%22%3A0.4117647058823529%2C%22a%22%3A1%7D%2C%22strokeEnabled%22%3Afalse%2C%22strokeWidth%22%3A0.8%2C%22strokeColor%22%3A%7B%22h%22%3A52.000000000000234%2C%22s%22%3A1%2C%22l%22%3A0.5%2C%22a%22%3A1%7D%2C%22shadowEnabled%22%3Atrue%2C%22shadowOffsetX%22%3A0%2C%22shadowOffsetY%22%3A100%2C%22shadowBlurRadius%22%3A0%2C%22shadowColor%22%3A%7B%22h%22%3A55.36363636363636%2C%22s%22%3A0.9401709401709404%2C%22l%22%3A0.5411764705882353%2C%22a%22%3A1%7D%2C%22shadowGradientEnabled%22%3Atrue%2C%22shadowGradientSteps%22%3A22%2C%22shadowEndColor%22%3A%7B%22h%22%3A37.42857142857143%2C%22s%22%3A0.9130434782608696%2C%22l%22%3A0.5490196078431373%2C%22a%22%3A1%7D%7D">pasting a link</a>.</p><p>CSS styling text is still more limited than box styles, but with some creative coding it is possible to generate outlines, fading 3d effects, and blurry fire. After doing a ton of research from great CSS design sites, I included some presets to show you the possibilities.</p><p>Finally, in addition to giving you the CSS for the style you’ve generated, you can also export a PNG with only the characters you care about using the Export PNG tab.</p><img src="https://joshondesign.com/images2/png-style-export.png" alt="z"/><p>My re-creation of CSS styles in Canvas isn&#x27;t perfect, but it&#x27;s pretty good. I hope this tool will be useful to you. The <a href="https://github.com/joshmarinacci/css-texteffects-builder">source is available on GitHub</a> in case you’d like to fork it. Feel free to contact me with feature requests and bug reports.</p>Fonts and Icons on Circuit Pythonhttp://joshondesign.com/2023/06/25/cp_font_conversionhttp://joshondesign.com/2023/06/25/cp_font_conversionSun, 25 Jun 2023 18:37:12 +0000<p>I’m continuing to work on <a href="https://joshondesign.com/2023/06/08/pico_round_lcd">a little smartwatch prototype</a> using a little round LCD and I want to have nice looking text. The default font for CircuitPython is fine, but it’s very tiny. This LCD has a pretty high DPI compared to other hobbyist screens( <em>&gt; 200 ppi</em>), so I need to find a new font. CP has a way of importing new fonts, but there are a few pitfalls and tricks I discovered, so that’s what I’m covering today.</p><p>Adafruit has <a href="https://learn.adafruit.com/custom-fonts-for-pyportal-circuitpython-display/bitmap_font-library">a guide to using custom fonts</a> on CircuitPython. CP only supports bitmap fonts, so you’ll need a font converted to BDF (ascii) or PCF (binary) format. Modern fonts are vectors instead of bitmaps, usually in TrueType (TTF) or related font formats, so we&#x27;ll need a converter. The command line program <a href="https://learn.adafruit.com/custom-fonts-for-pyportal-circuitpython-display/use-otf2bdf"><code>otf2bdf</code></a> is such a converter. If you’ve already downloaded a TrueType font locally you can convert with these instructions. I’m using MacOS but similar should work on Windows and Linux. </p><p>First install <code>otf2bdf</code> with brew, apt-get, or the install program of your choice</p><pre><code><code>brew install otf2bdf</code></code></pre><p>Now convert the font to bitmap at a specific point height. I’m using 16pt.</p><pre><code><code>otfbdf SomeFont.ttf -o cpfont.bdf -p 16</code></code></pre><p>Now copy the output bitmap font to your CircuitPython device and use it in Yython code like this:</p><pre><code><code>from adafruit_bitmap_font import bitmap_font<br/>from adafruit_display_text.bitmap_label import Label<br/>font = bitmap_font.load_font(&quot;cpfont.bdf&quot;)<br/>label = Label(<br/> font=font,<br/> text=&#x27;Text in a nice Font&#x27;,<br/>)</code></code></pre><p>And that’s it. Pretty easy.</p><h3>Icon Fonts</h3><p>Now let’s suppose you want some nice icons. You could download each icon as PNG or SVG and then convert to CircuitPython’s preferred BMP format, but icon fonts are a thing. Let’s use them. Google created the <a href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:opsz,wght,FILL,GRAD@48,400,1,0">Material icon font</a> for use across all of Android and the Web. They are completely open source and have icons for pretty much everything. We can’t use it directly, however, because it’s in what’s known as a <strong>variable webfont</strong> format which can have multiple weights and styles in a single file. Instead we want the static truetype fonts from the <a href="https://github.com/google/material-design-icons/tree/master/font">GitHub repo</a>. I chose the font called <em>MaterialIcons-Regular.ttf</em>.</p><p>When you download a single Material icon font you’ll see that it is huge. <strong>348KB for a single weight! That’s because it contains over 3000 icons.</strong> 348KB is too big for our little RP2040 that has less than 200KB, and of course we don’t need all 3000 icons. In most cases we only want five or ten of them. So we need a way to subset the font.</p><p>We will use a python program called pyftsubset that knows how to subset fonts. It is part of the <a href="https://fonttools.readthedocs.io/en/latest/index.html">font-tools</a>. Install font tools with pip then run pyftsubset to make sure it’s installed correctly.</p><pre><code><code>pip install fonttools<br/>pyftsubset --help</code></code></pre><p>Now let’s create a new icon font with a subset of the Material Symbol Font containing only the glyphs we want. If you select an icon back on the <a href="https://fonts.google.com/icons">Material Symbol Font webpage</a> a sidebar will appear on the right. At the very bottom it will list the <em>code point</em> of that particular icon. That’s what you need. Write down this code point, and the points of any other icons you want to extract. For my example I want the icons for Settings (e8b8), Battery Full (e1a4), and Timer (e425).</p><pre><code><code>pyftsubset MaterialIcons-Regular.ttf --unicodes=&quot;U+e8b8,e1a4,e425&quot; --output-file=icons.ttf<br/></code></code></pre><p>You can add the <code>--no-ignore-missing-unicodes</code> option to print an error if you typed in a code that doesn’t exist in the font. The resulting font is still in TrueType format, so we need to convert to bitmap to use it.</p><pre><code><code>otf2bdf icons.ttf -o icons.bdf -p 16</code></code></pre><p>And now we can use it in our program. To use the icons we have to specify the codes for each icon with unicode escapes:</p><pre><code>time_label = Label(<br/> font=icons,<br/> text=&quot;\ue8b8\ue1a4\ue425&quot;,<br/>)</code></pre><p>Here it is on my prototype device.</p><img src="https://joshondesign.com/images2/IMG_5631_small.jpeg" alt="z"/><p><em>C’est magnifique</em> </p>Performance Limitations of CircuitPython's DisplayIO Graphicshttp://joshondesign.com/2023/06/12/display_io_perfhttp://joshondesign.com/2023/06/12/display_io_perfMon, 12 Jun 2023 21:23:01 +0000<p>For my next project with the WaveShare Round LCD I want to create an animation of<a href="https://marvelcinematicuniverse.fandom.com/wiki/Eye_of_Agamotto"> Dr Strange&#x27;s Eye of Agamotto</a>. Here is my <a href="https://apps.josh.earth/waveshare_lcd_test/eye.html">first attempt in JS</a> (looks more like the comics than the movies). Not bad, I think.</p><img src="https://joshondesign.com/images2/agamoto-anim-screenshot.png" alt="z"/><p>When it animates the eye lids come down. I built them with quadradic curves that change over time using Canvas. Canvas is fast enough to let me make the animation very smooth. Once porting it to CircuitPython on an embedded device, however, well.. it&#x27;s pretty damn slow.</p><h3>Animation in CircuitPython</h3><p>For my first attempt I captured the JS animation as a video, scaled it down to 240 x 240, then converted it to an animated gif. On the device the gif renders at 2 or 3 fps. A far cry from the smooth 60fps of my browser. </p><p>For my second attempt I exported a background image of the disk, drew the eye on top, then animated the eye lids as rectangles that move from the top and bottom to the center. It was still rough, but seemed to be maybe 6 or 7 fps.</p><p>What&#x27;s going on here? While this device is slow, it&#x27;s stil a 133mhz process. I had far faster animation back on my old pentium, even when running Java code instead of optimized C. The problem must be in how the display is updated. After spending the next 8 hours researching how these sorts of displays work, and how CircuitPython exposes them to the programmer, I&#x27;ve come to this conclusion: <strong>Tiny TFT displays are just slow by nature, even with raw C code, but there are ways we can speed it up.</strong> That&#x27;s what the rest of this post is about.</p><h3>How CircuitPython Does Graphics</h3><p>CircuitPython can be quite fast because all of the performance sensitive work is implemented in optimized C code. This includes drawing. The displayio system lets you use bitmaps with indexed colors to save memory, which means copying bitmaps back and forth can be fast. Drawing shapes can be almost as fast as C because we are just setting pixels in an in memory bitmap, and of course things like vertical lines and rectangles can be optimized using memcpy routines. Actually drawing to the screen is a different story. It can be very slow.</p><p>No matter how you draw into the framebuffer, that buffer has to be uploaded to the display a bitmap. Most of the little hobiest displays use a serial bus called SPI. Looking at the source to the driver for my board, the only thing custom was the init code. All uploading is a standard FourWire connection, which is a type of SPI connection. </p><p>I’m using a 240x240 16bit color screen connected over SPI. It has no indexed modes or accelerated drawing routines (as far as I can tell none of these hobbiest screens do). This means to do a full screen refresh requires sending two bytes for every pixel on the screen, over a serial port, one bit at a time. The DisplayIO routines are smart and try to use batches, but even if you set the entire screen to a single color, that is still <strong>a lot of data to transfer</strong>.</p><p> 240 x 240 x 2 bytes per pixel equals 115200 bytes per screen, or <strong>912000</strong> <strong>bits per frame</strong>. For 30fps that’s <strong>over 27 million bits per second with zero overhead.</strong> In practice I’m guessing we get half of that. A lot of SPI ports only run at 10 or 20 MHz, so getting a consistent 30fps just isn’t going to happen. That’s just the nature of SPI. Desktop computer screens use protocols that can send more than one bit at a time, and operate much faster than 10Mhz. SPI just can&#x27;t.</p><p>There is one saving grace, however. These screens have an internal memory buffer and <strong>support partial refreshes</strong>. This means you can update rectangular subsections of the screen containing <strong>just the changes</strong> between the last frame and the next. It won’t help for full screen animation, but for typical GUI work where only a few things change we can make it quite performant. We just have to go back to rendering algorithms from the 90s (like the Java Swing toolkit that I worked on for 10 years). This is in fact what <em>displayio</em> does.</p><p>The inability to do smooth full screen refreshes does mean certain types of effects, like slideshow animations and or fading the entire view to black are impossible. (Well, maybe we could animate the backlight?)</p><p>Earlier I said there are not acceleration commands, like RLE encoding or bitblts, or indexed color palette swaps. This is true. However, this particular chip does have vertical scroll offsets, which might make it possible to do smooth vertical scrolling of at least part of the screen. I’ll have to look into that later.</p><h3>Improving Performance</h3><p>So to start with, the only perf improvement we can do is set the SPI bus to the fastest possible speed. In the sample micropython code that comes with this screen I *think* they are setting the speed to 100mhz but the C code seems to be 40mhz and the Arduino code 66mhz. 🤷 Switching from 20mhz to 100mhz seems to speed up the animation I’m working on, but really I need a proper benchmark to measure it.</p><p>I created a little script to redraw a fullscreen bitmap over and over, with different SPI speeds. At 1mhz I’m getting an 690ms per frame or about 1.45fps. As I increase the speed the fps improves linearly until <strong>it tops out at 64 MHz for a frame time of 144ms or just under 7fps</strong>. Setting the speed any higher has no effect. Calculating the multiplying the size of a frame in bits times the fps gives me 6.4Mbps, which sounds correct for an uncompressed video stream. This seems to be the max we can get.</p><p>Incidentally if I leave off the speed setting it seems to default to 20Mhz. If I leave off the target framerate of 60 it drops to 3.66fps. If I set the target framerate to 10 (which is still higher than our max) it drops to 5.38fps. Setting the target to 60 lets get back to our max fps of 7. So I’m not sure what exactly the target framerate is doing but it seems to have some inaccuracies. Perhaps it’s allowing for more GCs? Setting it to 100 actually seems to increase the fps slightly to 7.09. Setting it to 1000 increases it some more, but only to 7.3. So clearly there is some tradeoff here but if it’s impacting battery or heat then it’s not worth it for that tiny speed improvement.</p><h3>Partial Refresh</h3><p>Now let’s try filling just part of the screen and see if we can get speedups?</p><p>240x240 = 7.09 fps</p><p>240x120 = 13.89 fps</p><p>120x120 = 25.48 fps</p><p>Sure enough: <strong>halving the pixels roughly doubles the frame-rate</strong>. Cutting them to a quarter gives us a 4x speedup. So 25fps seems pretty good, right? Yes and no. The numbers say it is pushing that many frames per second, but visually it does not seem to actually be rendering that fast. In fact there is a lot of tearing. Recording with my slow-mo camera shows that it takes about 70-100ms to render a frame, or in the range of 10-14fps, even though we are sending significantly more frame data than that. Technically we could refresh at 64x64 rectangle over 60fps, assuming we had no overhead, but that’s only 16th the number of pixels. And visually it doesn’t look that fast, plus processing that much data in code for something like a video would be insane.</p><p>Still, this does give us something to work with. A full screen refresh is around 7fps, but if we only want to refresh part of the screen at a time then we can get 15 to 30fps fairly easily. </p><p>Let’s try doing several circles rotating. I can get a calculated fps of over 30 fairly consistently. In fact, if I set the target framerate to 20 and the speed to 200Mhz I get an extremely consistent exactly 32fps. Something must be syncing up nicely here. I&#x27;m not doing anything clever in my code. I created some circles and move them every frame. DisplayIO must be doing some dirty tracking underneath and only sending the smallest number of changed pixels to the screen. That&#x27;s how we get 30fps. But again, visually it does not seem as smooth and there is some tearing. I suspect the refresh rate of the physical LCD is lower than how fast we can pump out screens. It also looks a little choppy because my circles are restricted to integer coordinates. I can&#x27;t move a circle to be on the edge of two pixels and get anti-aliased drawing like I would on HTML canvas. But for what I&#x27;m building I can live with these constraints.</p><h3>Next Time</h3><p>So what have we learned?</p><ul><ol><li>All of my research taught me that the SPI bus is the bottleneck, and there&#x27;s no way to get more than 7fps for a full screen refresh. However, partial screen refreshes can quite fast and the displayio API was designed to enable this. They&#x27;ve done a great job. I don&#x27;t think I could do better without hardware changes.</li><li>Make sure you set the SPI bus to it&#x27;s max speed. The default may be much lower than the component is capable of.</li><li>Design your graphics an animation around updating the fewest number of pixels</li><li>Use the backbuffer to your advantage. You can make complex graphics as long as the number of pixels changed per frame is small. Cool particle effects should be possible with this technique.</li></ol></ul><p>Next time I&#x27;ll show you some graphics examples I&#x27;ve been working on as well as how to access the touch events to make a little painting app. All of the source code for this project, including my performance testing on in <a href="https://github.com/joshmarinacci/waveshare_lcd_test">this github repo</a>.</p>Playing with a Pico based Round LCD screenhttp://joshondesign.com/2023/06/08/pico_round_lcdhttp://joshondesign.com/2023/06/08/pico_round_lcdThu, 08 Jun 2023 20:57:00 +0000<p>I&#x27;m enamored of all of cheap and hackable screens and embedded CPUs coming onto the market. I&#x27;m frustrated, however, that despite embedded computing being more accessible than ever, the mass manufacturers products are all becoming more and more the same. Where are the phones with oval screens? Where are the steam punk pocket calculators? We have the technology to make devices that are ever more niche and increasingly creative and different, yet what you can actually buy is more glowing rectangles. I&#x27;d like to fix this.</p><img src="https://joshondesign.com/images2/waveshare_rp2040_lcd_1_28.jpg" alt="z"/><p>Recently I found this cool micocontroller with a <strong>round</strong> LCD made by Waveshare. It&#x27;s built around the RP2040 chip that sits at the heart of the Raspberry Pico, making it very compatible with Arduino and Python. It has some built in sensors, 4MB of flash, and lipo charger circuit. I ordered one last week and it just arrived. Let&#x27;s dive in.</p><p>Though the <a href="https://www.waveshare.com/wiki/RP2040-Touch-LCD-1.28">official website</a> only mentions C and MicroPython support, there is a beta CircuitPython firmware build available to download <a href="https://circuitpython.org/board/waveshare_rp2040_lcd_1_28/">here</a>, which is what I&#x27;ll be using today. I also found <a href="https://crates.io/crates/waveshare-rp2040-lcd-0-96">a Rust create for a related variant</a> of the board, so I might try hacking on that later.</p><h3>First boot</h3><img src="https://joshondesign.com/images2/waveshare_pico_128LCD.jpeg" alt="z"/><p>After plugging it in the device shows a boot screen, photo of some greenery, and then this live sensor output screen; highlighting the built in accelerometer, gyroscope, and battery sensor.</p><p>In the photo you can see just how small this thing is. <strong>1.28 inches across!</strong> That&#x27;s a USB-C connector at the bottom there. The screen has a 240x240 pixel display that I found to be pretty sharp. It looks a little blurry here because I hven&#x27;t taken off the plastic protective film yet.</p><p>On the back there are two buttons for <strong>boot</strong> and <strong>reset</strong>.<strong> </strong>When I plugged it into my computer it did not come up as a local drive. When I held down <strong>boot</strong> while plugging it in my Mac asked to trust the device and a tiny drive called <strong><em>RPI-RP2</em></strong> popped up. The bootloader text shows:</p><pre><code><code>UF2 Bootloader v3.0<br/>Model: Raspberry Pi RP2<br/>Board-ID: RPI-RP2</code></code></pre><p>So far so good. After dragging on the CircuitPython UF2 file to the device it rebooted and came up as a <strong>CIRCUITPY </strong>drive. Using <code>screen /dev/tty.usbmodem2101</code> got me to the python shell. Success!</p><h3> Drawing to the Screen</h3><p>I struggled to draw to the screen for a few hours. Some research indicated this device uses the <strong>gc9a01</strong> graphics driver, which is supported by the latest CircuitPython release. None of the sample code I found worked, however. The backlight would come on but nothing was drawn. By coomparing to older <a href="https://gist.github.com/aallan/8f22a32c73696fe61c4c43d00063faa4">MicroPython code by Alasdair Allan</a>, I confirmed I was using the correct pins. Even the <a href="https://www.waveshare.com/wiki/RP2040-LCD-1.28">main spec page</a> says I am.</p><p>After some more searching I came across <a href="https://www.waveshare.com/wiki/RP2040-Touch-LCD-1.28">this other data page</a> specifically for the touch version. Nothing seemed awry, but looking at <a href="http://cdn.static.spotpear.com/uploads/picture/learn/raspberry-pi/rpi-pico/rp2040-lcd-1.28-touch/RP2040-Touch-LCD-1.28.pdf">the schematic diagram</a> I could see the pins are slightly different than the non-touch version, <strong>and</strong> different from the docs in the demo code. </p><p>In the end having the touch version meant one pin was different. Below is the code I ended up using. Notice that reset is not set to <code>LCD_RST</code> as would be on the non-touch device. Instead we have to use pin 13.</p><pre><code>spi = busio.SPI(clock=board.LCD_CLK, MOSI=board.LCD_DIN)<br/># LCD_RST is 12 in regular version<br/># but we need 13 for the touch version<br/>display_bus = displayio.FourWire(spi, <br/> command=board.LCD_DC, <br/> chip_select=board.LCD_CS,<br/> reset=board.GP13)<br/>display = gc9a01.GC9A01(display_bus, <br/> width=240, <br/> height=240, <br/> backlight_pin=board.LCD_BL)</code></pre><p>And with that everything worked. Here&#x27;s a photo of the LCD showing a indexed bitmap of Earth.</p><img src="https://joshondesign.com/images2/running_earth_lcd.jpeg" alt="z"/><h3>Next Time</h3><p>Now that I have the board working it&#x27;s time for some fun. Next up I want to get print a case for it, access the accelerometer, and build some sort of interaction app. Maybe a puzzle game.</p>Canvas Scaling and Smoothing Trickshttp://joshondesign.com/2023/04/15/canvas_scale_smoothhttp://joshondesign.com/2023/04/15/canvas_scale_smoothSat, 15 Apr 2023 17:06:27 +0000<p>I love HTML Canvas. I even <a href="https://joshondesign.com/p/books/canvasdeepdive/title.html">wrote a book about it</a> once. Canvas is good drawing API and it runs everywhere. However, despite the magic that is the Canvas API, it can still be tricky to use when it interacts with CSS. I often have people ask me <em>how to make their canvas fill the screen</em>, or resize <em>with the window</em>, or to have a <em>fixed aspect ratio</em> but still scale to fit inside the window without overlap. All of these require understanding some internal details about how Canvas works. So let’s dive in.</p><p>Canvas is a <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/Replaced_element"><em>replaced element</em> </a>in HTML. This means it is a rectangle with a <em>fixed internal size</em>. Essentially the browser treats a Canvas element as an Image. This means that some CSS styles meant for images <em>also work for Canvas!</em> You can tell the canvas to be a certain pixel size and use CSS to change its visual size. For example you could make a canvas that is 256x256 pixels, then scale it up to be 2560x2560. The canvas doesn’t know the difference; it draws into its buffer just like it was an image.</p><p>In the case below a 256px magenta circle on a white square canvas that is resized to be 8000 pixels across. The <code>height</code> is <code>auto</code> so it will maintain the same aspect ratio, just like an image. </p><pre><code><code>&lt;canvas width=&quot;256&quot; height=&quot;256&quot;&gt;&lt;/canvas&gt;<br/>&lt;style type=&#x27;text/css&#x27;&gt;<br/> canvas {<br/> width: 8000px<br/> height: auto<br/> }<br/>&lt;/style&gt;<br/></code></code></pre><p>A really big image</p><img src="https://joshondesign.com/images2/super-zoomed.png" alt="z"/><h3>Adapting for High DPI Screens</h3><p>Now let’s consider HiDPI. Suppose you want a 500 x 500 pixels canvas, but if the viewer has a display with 2x HiDPI support then it should scale it up to look the same, but sharper. We can do this by increasing the pixels in the canvas and using CSS to constrain it back to its 500 x 500 pixel size. Remember that in CSS pixels are <em>virtual pixels</em>. They are scaled by the browser automatically. So on a 2x screen we’d want the canvas to be 1000x1000 image pixels but scaled back to 500x500 virtual pixels, which will be scaled back up by the browser to 1000x1000 physical screen pixels.</p><pre><code><code>&lt;canvas id=&quot;can&quot; width=&quot;1000&quot; height=&quot;1000&quot;&gt;&lt;/canvas&gt;<br/>&lt;style type=&#x27;text/css&#x27;&gt;<br/> canvas {<br/> width: 500px;<br/> height: auto;<br/> }<br/>&lt;/style&gt;<br/>&lt;script type=&#x27;javascript&#x27;&gt;<br/> const ctx = document.getElementById(&#x27;can&#x27;)<br/> ctx.fillStyle = &#x27;red&#x27;<br/> ctx.beginPath()<br/> ctx.arc(250,250,250,0,Math.PI*2)<br/> ctx.fill()<br/>&lt;/script&gt;<br/></code></code></pre><p>Left is without the fix, right is with the fix.</p><img src="https://joshondesign.com/images2/with-and-without-fix.png" alt="z"/><p>Notice in this zoom how the pixels are sharper in lower image</p><img src="https://joshondesign.com/images2/zoomed-in.png" alt="z"/><p>Of course the ratio might not be 2x. There are some mobile devices with 3x screens. Also, the canvas now thinks it has 1000px of space, but if your drawing code assumes 500px it will will only draw in the upper left corner of the canvas. Let’s scale the size of the canvas to use the real device’s DPI in code, and then reverse the scale for drawing the circle. The property <code>window.devicePixelRatio</code> will give us the correct scaling factor.</p><pre><code><code>&lt;canvas id=&quot;can&quot; width=&quot;500&quot; height=&quot;500&quot;&gt;&lt;/canvas&gt;<br/>&lt;script type=&#x27;javascript&#x27;&gt;<br/> const canvas = document.getElementById(&#x27;can&#x27;)
<br/> let dpi = window.devicePixelRatio<br/> const WIDTH = 500<br/> const HEIGHT = 500<br/> canvas.width = WIDTH*dpi<br/> canvas.height = HEIGHT*dpi<br/> canvas.style.width = `${WIDTH}px`<br/> canvas.style.width = `${HEIGHT}px`<br/> const ctx = canvas.getContext(&#x27;2d&#x27;)<br/> ctx.save()<br/> ctx.scale(dpi,dpi)<br/> ctx.fillStyle = &#x27;white&#x27;<br/> ctx.fillRect(0,0,WIDTH,HEIGHT)<br/> ctx.beginPath()<br/> ctx.arc(WIDTH/2,HEIGHT/2, WIDTH/2,0, Math.PI*2)<br/> ctx.fillStyle = &#x27;magenta&#x27;<br/> ctx.fill()<br/> ctx.restore()<br/>&lt;/script&gt;<br/></code></code></pre><h3>Preserving Chonky Pixels</h3><p>We’ve seen that scaling up a canvas to handle high resolution is easy. But what if we need the opposite? Suppose we were making a game with retro-style pixel art then we <em>want</em> to see the chunky pixels. Unfortunately if we just scale it up the pixels get blurry, like this:</p><img src="https://joshondesign.com/images2/bad-blurry-pixels.png" alt="z"/><p>To fix this we need to tell the browser to not smooth the pixels. CSS already has a property for this, <em>image-rendering: pixelated</em> and it works for canvases too.</p><p>With that one css property it looks like this: </p><img src="https://joshondesign.com/images2/good-chonky-pixels.png" alt="z"/><p>Note that if you draw <em>scaled</em> images inside the canvas they may still be smoothed. You can fix this by setting <code>imageSmoothingDisabled:true</code> on the drawing context</p><img src="https://joshondesign.com/images2/bad-blurry-image.png" alt="z"/><img src="https://joshondesign.com/images2/good-chonky-image.png" alt="z"/><h3>Big but not too big</h3><p>Now that we have a retro game canvas, say a 160 x 120 pixel screen, we might want it to grow as big as possible to fill the extra space on page. We could use <code>width:100%</code> but the bottom might get cut off if the window is wide but short. Setting both <code>width</code> and <code>height</code> to <code>100%</code> would make it fit but stretch and squish the pixels. Instead we want the canvas to be as big as possible while maintaining it’s aspect ratio, but not so big that any part of it is hidden. This turns out to be a common need for images as well, so there’s already a CSS property that does it: <code>object-fit.</code> With <code>object-fit:contain</code> the canvas will shrink to be completely <em><strong>contained</strong></em> within the styled size.</p><p>In this example a 160x120px canvas is stretched to 500x500.</p><pre><code><code>canvas {<br/> width:500px;<br/> height:500px;<br/> image-rendering: pixelated;<br/> background-color: black;<br/>}</code></code></pre><img src="https://joshondesign.com/images2/object-fit-none.png" alt="z"/><p>If we set <code>object-fit</code> to <code>contain</code> then we get this, with black bars (the background of the canvas) filling the extra space.</p><img src="https://joshondesign.com/images2/object-fit-contain.png" alt="z"/><p>And if we set <code>object-fit</code> to <code>cover</code> it will expand to fill all space, but possibly crop the edges off. Not good for our retro game but it could be useful for other cases where you don&#x27;t know the rendered size until the page is loaded.</p><img src="https://joshondesign.com/images2/object-fit-cover.png" alt="z"/><p>The <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/object-fit">object-fit - CSS: Cascading Style Sheets | MDN</a> specs page describes all of the possible values. This css property is really for images, but remember: anything an image can do a canvas can do better!</p><h3>canvas to fill the screen</h3><p>So far we have used a canvas with a fixed size. We might scale the size up, but it has a definite size that doesn’t depend on the size of the browser window or any content next to the canvas. Sometimes we <em>do</em> want the canvas to adapt to its surroundings. Consider a drawing tool. It needs to resize when the user resizes the window. We could give the canvas a width and height of 100%, which would make it stretch to fill the available space, but, just like an image, that no longer preserves the aspect ratio. The pixels would get squished. Instead we need the canvas to change both its layout (<em>css pixels</em>) and its drawing surface (<em>image pixels</em>) to handle the current conditions. To do this we’ll need some code.</p><p>First we need to figure out how to get anything to fill the page. Let’s start with a <code>div</code> wrapping a <code>canvas</code>. Setting the width and height of the div to 100% does this:</p><img src="https://joshondesign.com/images2/resize-v1.png" alt="z"/><p>Hmm. Not what we want. The problem is that the <code>body</code> element is special. It always takes up the full width of the window’s viewport, but the <em>height</em> depends on the content. What we want is for height to be exactly 100% of the viewport’s height. Enter CSS vh and vw units. Setting the div&#x27;s size like this</p><pre><code>div {<br/> width: 100vw;<br/> height: 100vh;<br/>}</code></pre><p>gives us this</p><img src="https://joshondesign.com/images2/resize-v2.png" alt="z"/><p>Depending on your browser you might get some scroll bars. This just means there’s extra space we need to get rid of. Set the <code>padding</code> and <code>margins</code> to 0 for both the <code>div</code> and <code>body</code>, and set <code>box-sizing</code> to <code>border-box</code> to account for the 5px colored borders. Now we get <em>this</em>, and resizes properly with the window.</p><img src="https://joshondesign.com/images2/resize-v3.png" alt="z"/><p>Now that the div is sizing properly, we just need to update the canvas itself when the window resized. Again we will use a little bit of code. On every size it will set the canvas size to the size of the div, minus the borders. Depending on your browser you might also need to set <code>overflow: hidden;</code> on the <code>body</code> element.</p><pre><code> function calcCanvasSize() {<br/> // get the wrapper size<br/> const wrapper = document.getElementById(&#x27;wrapper&#x27;)<br/> let rect = wrapper.getBoundingClientRect()<br/> const can = document.getElementById(&#x27;can&#x27;)<br/> // set the canvas size to wrapper, minus the 5px borders<br/> can.width = rect.width-5*4<br/> can.height = rect.height-5*4<br/> const WIDTH = can.width<br/> const HEIGHT = can.height<br/> // redraw<br/> const ctx = can.getContext(&#x27;2d&#x27;)<br/> ctx.fillStyle = &#x27;white&#x27;<br/> ctx.fillRect(0,0,WIDTH,HEIGHT)<br/> ctx.beginPath()<br/> ctx.arc(WIDTH/2,HEIGHT/2, WIDTH/2,0, Math.PI*2)<br/> ctx.fillStyle = &#x27;magenta&#x27;<br/> ctx.fill()<br/> }<br/> calcCanvasSize()<br/> window.addEventListener(&#x27;resize&#x27;,calcCanvasSize)</code></pre><p>And now we get a beautifully resizing canvas.</p><img src="https://joshondesign.com/images2/resize-v5.png" alt="z"/><h3>Conclusion</h3><p>I hope today you&#x27;ve seen how easy it is to style the canvas to different uses. As browsers introduce more css properties for images, almost all of them will work for canvas elements. Go Canvas!</p><p>Code for everything in this blog is available on this <a href="https://github.com/joshmarinacci/canvas-blog">git repo</a>.</p>One Hour Ponghttp://joshondesign.com/2023/01/12/one_hour_ponghttp://joshondesign.com/2023/01/12/one_hour_pongThu, 12 Jan 2023 23:03:25 +0000<p>I’ve been teaching my nephew to code for the past few years and we did a game jam together over the winter break. He said to me: “Uncle Josh, you sure can code fast.” I reminded him that I’ve been programming for 30+ years, and one day he’ll be even better than I am. </p><p>My nephew struggles to come up with ideas and often goes down rabbit holes on one feature or tool instead of working on the rest of the game. I told him that time constraints can make a game better because it forces you to focus. Ever the clever child, he suggested I do <em>a pong game in one hour</em>. <strong>From scratch</strong>. <em><strong>With no existing code or game frameworks</strong></em>. Challenge accepted!</p><p>Here’s what I built last Tuesday.</p><img src="https://joshondesign.com/images2/one_hour_pong.png" alt="z"/><p><a href="https://apps.josh.earth/one_hour_pong/">Live Demo</a></p><h3>Construction </h3><p>To build the game I first spent about 15 min doing setup. Point and Bounds classes. Ball, Paddle, Game state. And of course a game loop with functions for input, collisions, draw, check for game over. However, I didn’t actually write these functions or classes yet. I just stubbed in the names and left the implementations for later. My nephew asked why I was doing it this way. “Having the stubs helps to organize your brain and your code. I’ll fill in the code as I need it. That way you don’t wast time building parts that you don’t end up needing.”</p><p>By the 20 minute mark I had basic drawing working, and movement with keyboard inputs by 25 min. Collisions and physics ended up being the hardest part. I wrote a <code>Bounds</code> class with an <code>intersection</code> method, but that wasn’t enough. To do collisions correctly the direction of the ball&#x27;s motion matters. I ended up doing a more brute force method where I check the ball against the bottom of the top wall, the top of the bottom wall, each paddle, etc. Not as clean as I’d like, but far simpler.</p><p>I managed to add sound and fade effects in the last 5 min. With 30 seconds remaining I remembered I still needed to make it &quot;pretty&quot;, which ended up being just changing colors to a clean grayscale with red for the ball.</p><p>Once the challenge was over I did spend another 20 minutes fixing the fading code. It worked before but was wrong. I also tweaked the size of the canvas, added wall bumpers, and a click event to trigger audio. (I always forget that part).</p><p>What you see above is with the extra 20 min. </p><h3>Tools used:</h3><ul><li><a href="https://www.typescriptlang.org">Typescript</a>. Using Typescript with a good IDE (<a href="https://www.jetbrains.com/webstorm/">WebStorm</a>) can be magical.</li><li>For sound effects I used the JSFXR <a href="https://sfxr.me">web tool</a> and embedded it in the game using a string encoding instead of downloading wave files.</li><li>I started using BeepBox to create a music composition for the background, but ran out of time. You can hear the <a href="https://www.beepbox.co/#9n31s0k0l00e03t2ma7g0fj07r1i0o432T1v1uc0f10l7q011d23A4F3B5Q0506Pd474E361963279T0v1u58f0q0x10ob1d03w5h1E1b7T1v1u3df0qwx10p511d08AcFbBfQ269cP969bE2bi7iT4v1uf0f0q011z6666ji8k8k3jSBKSJJAArriiiiii07JCABrzrrrrrrr00YrkqHrsrrrrjr005zrAqzrjzrrqr1jRjrqGGrrzsrsA099ijrABJJJIAzrrtirqrqjqixzsrAjrqjiqaqqysttAJqjikikrizrHtBJJAzArzrIsRCITKSS099ijrAJS____Qg99habbCAYrDzh00E0b4h400000000h4g000000014h000000004h400000000p1WBWqfibSqfVgzjhWhvgnVBpp60BWqfijtfMs600aqcMnQ5Z17ghQ4t5B960">little bit I managed to make here</a>.</li></ul><h3>Conclusion.</h3><p>Doing a one hour game was really fun. The time constraint pushed me to think fast and balance planning with YAGNI. I’m really impressed with how far you can get with modern canvas and javascript, no frameworks required.</p><p>You can view <a href="https://github.com/joshmarinacci/one_hour_pong">the soruce here.</a></p>Why you can’t build a web browser and why you should anyway.http://joshondesign.com/2022/12/14/browser_1000_lochttp://joshondesign.com/2022/12/14/browser_1000_locWed, 14 Dec 2022 16:01:16 +0000<p>In the last couple of years I’ve seen a lot of <a href="https://www.marabyte.com/post/on-browser-monoculture/">lamenting</a> about <a href="https://midrange.tedium.co/issues/monocultural-studies/">the browser mono-culture</a>. I even <a href="https://joshondesign.com/2022/04/14/browser-engine-monoculture">wrote about it myself</a>. Some complains focus on <a href="https://seldo.com/posts/are_we_making_the_web_too_complicated">how complicated</a> the web specs <a href="https://www.reddit.com/r/webdev/comments/x5sra9/web_development_is_so_complicated/">have become</a>. So big that only a few companies can implement a browser from scratch. I think these complaints are misplaced. Even if the web platform didn’t have such a large API surface it still wouldn’t matter. You can’t build a large scale browser with large marketshare. The browser market would still be a monoculture. You can&#x27;t solve a business problem with a technology solution. I also don&#x27;t think that replacing the web with something smaller like <a href="https://gemini.circumlunar.space">Gemini</a> is the answer. </p><p>But you know what? <strong>We should build new browsers anyway! And to prove it I just made one in less than 1000 lines of code!</strong> Let&#x27;s dive in.</p><h3>No One Can Build a Browser</h3><p><em>WebKit vs Chromium.</em> <em>Firefox is the last independent browser</em>. <em>Blah blah blah</em>. This is a lot of whining without getting to the root cause. Yes, no one can make a new browser anymore, but so what? No one but a few people invest in building a new browser because <strong>there is no motivation for it</strong> unless you control the OS or a search engine. <strong>There is no money in the browser itself</strong>. WebKit is driven by Apple&#x27;s Safari team which is driven by the selling of Mac hardware. Chromium is driven by the Chrome team which is driven by Google’s search engine. <em>Of course</em> no one can build a new browser. </p><p>Building a browser means you&#x27;re competing with <strong>two of the richest companies in the history of the world</strong>. <em>Of course</em> they are the only ones who can invest billions in a commodity product that has no direct revenue. Even Microsoft couldn’t do it; Edge is their Hail Mary Pass to not be displaced on their own platform. Even if somehow you could make a new full spec browser it <em>still</em> wouldn’t matter because you’d never get marketshare from the platform owners controlled by, again, some of the largest companies ever. Even if you win you’d lose.</p><p><em>Side note: When I was at Mozilla I urged senior management to switch the desktop browser to WebKit (to lower costs since no one cares what the engine is) and focus on bringing FF to new platforms (like standalone VR headsets).</em></p><h3>The Problems with a Browser Monoculture</h3><p>So yeah, no one can build a browser. No one can start from scratch and keep up with Apple &amp; Google. Even Microsoft couldn’t. But why does this matter?</p><p>First, it&#x27;s a problem because it hinders development of alternative operating systems. A new OS isn&#x27;t usable in the modern would without a full spec browser. It&#x27;s the same issue as launching an OS without an existing App Store. Embracing the smol web or creating new protocols won’t fix any of these issues. Building a new browser won&#x27;t help. This is a real problem, I agree. It does stop new platforms from emerging because the platform under you is always an intermediator. This is one reason why Facebook has invested so much in VR and doesn&#x27;t use the Chrome browser on the Quest.</p><p>We will never beat the browser makers at their own. Let’s play a different game instead. Let’s go back to the beginning and think smol</p><h3>Gemini?</h3><p>I like the idea of the SMOL web and Gemini. But I think their approach is wrong. The <a href="https://en.wikipedia.org/wiki/Gemini_(protocol)">Gemini protocol</a> throws the baby out with the bathwater. It doesn&#x27;t even allow anchors so you can jump to a specific part of a document. This was a feature of the very first version of the web back in the mythical golden days that the SMOL web longs for.</p><p>In the Gemini FAQ they say: <em> </em><a href="https://gemini.circumlunar.space/docs/faq.gmi"><em>Why not just use a subset of HTTP and HTML?</em></a></p><blockquote>The problem is that deciding upon a strictly limited subset of HTTP and HTML, slapping a label on it and calling it a day would do almost nothing to create a clearly demarcated space where people can go to consume *only* that kind of content in *only* that kind of way.</blockquote><p>I agree that this <a href="https://xn--gckvb8fzb.com/gemini-is-solutionism-at-its-worst/">is Solutionism at it’s worst</a>.The Gemini protocol is really beside the point. What matters is creating a part of the Internet where tracking and spam are not possible. The Gemini community is enforcing this with a protocol, but it’s the tracker-less part of the web that is the point. That&#x27;s where the value is.</p><p>I like Gemini&#x27;s goal, but it has problems. First, it’s exclusionary. By making it different it becomes harder to get there. The Gemini part of the Internet will always be insular and separate. Second, it doesn’t solve any real problems. It doesn’t address the reasons The Web became a cess pool of spam like this in the first place. Essentially Gemini is giving up on the world and saying<em> &quot;We&#x27;ll just move elsewhere and build our own tiny web without you!&quot;</em> Fine if you want that, but it’s awfully depressing and unproductive. I don’t think it actually helps anyone.</p><p>Ultimately Gemini isn’t really a technology or protocol, <strong>it’s a social statement.</strong> It’s about building a community that says <em>&quot;no tracking and spam is tolerated here&quot;</em>, and they are using the protocol to enforce this. I like it. I just think there’s more we could do that would reuse existing technology in better ways, and benefit more people.</p><h3>Let&#x27;s Build a Browser Anyway</h3><p>The Gemini FAQ claims that their spec is easy enough to implement in 50 to 100LOC. I don’t know if that’s a good metric. Certainly HTML &amp; CSS is more complicated, but it’s not <em>that </em>much more complicated. I&#x27;ve found that a lot of things that seem hard or impossible become solvable or even easy if you are willing to <em>relax your constraints</em>. </p><ul><li>A high performance Javascript VM: <strong>hard</strong>. A slow JS interpreter: <strong>easy</strong>.</li></ul><ul><li>A full spec compliant PDF parser. <strong>hard</strong>. A PDF 1.1 parser (which is the parts most people want), <strong>far easier</strong>.</li></ul><p>To prove this point I wrote a browser implementation in less than 1000 lines of code. (I was shooting for 500 lines, but currently I’m stuck around 750). Here’s what it looks like:</p><img src="https://joshondesign.com/images2/Screen Shot 2022-12-14 at 9.08.58 AM.png" alt="z"/><p>This is 750 LOC of typescript. The parsers for HTML and CSS are not spec compliant, but they do handle a useful subset and each is tiny! I mean tiny on the order of <a href="https://github.com/joshmarinacci/browser500/blob/main/src/style.ts">20 lines for the grammar</a> thanks to the magic of <a href="https://ohmjs.org">OhmJS</a>, a PEG toolkit. This prototype example was not meant to actually be a usable browser, but to prove that it can be done. </p><p>My mini browser really does parse and render HTML and CSS the way it would have been done 15 years ago. It has links, lists, paragraphs and images. It supports basic CSS properties and the cascade. It does not support <em>all</em> of the CSS spec, but it is written in such a way that it would be clear where you could add them. </p><p>Okay, I’m <strong>sort of cheating a bit </strong>because I did it in a browser, but it’s not that much of a cheat. The only thing I’m using from the host browser is the graphics and network layers, both of which a standalone app would leverage the OS for anyway, so I think it’s fair. The implementation is simple enough that I could see this ported to something like a Raspberry Pico (not a Pi, <a href="https://www.google.com/search?client=safari&amp;rls=en&amp;q=raspberry+pico&amp;ie=UTF-8&amp;oe=UTF-8">a Pico</a>) with a bit of supporting C/C++/Rust code.</p><p>And yes it has some rendering glitches. I said it works, not that it’s bug free. But this proves that in a thousand lines of code we can make something that can actually browse the web. And if I could get this far in less than a thousand lines of Typescript, <em><strong>what could be built with 10k of Rust code?</strong></em> You could have a very fast and lean browser that provides real value for a lot of people. It could even support images and video links *without* including Javascript support. (The audio and video elements are a thing for a reason).</p><h3>Uses for a smaller independent Web Browser engine</h3><p>Is it possible to make a new full browser that is so complete it can run Facebook or Google docs? No. We&#x27;d need to fork an existing engine for that. But, can you make a browser that is useful? <strong>Yes</strong>! There’s lots of the web you can render. The web standards are designed to degrade gracefully. Just because most pages use JS doesn’t mean the browser has to render it. The Web !== React SPAs. There’s so much more.</p><ul><li>A new lightweight browser that’s a full native desktop app with zero JS support. It would be *fast* and use very little memory. It could even run on retro computers.</li><li>A replacement for Electron that includes the styling and layout parts of the web without the mess that overhead of a second runtime.</li><li>a library for building News and feed readers. </li><li>In game rendering of 2d rich text and images (think catalog and in game news updates)</li><li>PDF generation</li><li>local app development. Imagine how much faster Electron could be if it didn’t need to support every possible browser API?</li><li>offline bookmarks reader</li><li>eBook renderer</li><li>web browser or UI for tiny embedded systems like a Raspberry Pico</li><li>A knowledge archive app</li></ul><p>Such a browser should focus on speed and new uses. It should focus on good experiences. When you need a real browser for something, then just open a call out to the user’s preferred full browser.</p><h3>The Web Scales Down Too</h3><p>We must remember that <strong>the web was designed to scale.</strong> Just because the spec details it doesn’t mean you have to implement it. From the very beginning <em>graceful degradation</em> was a key value point of The Web. You <strong>can</strong> make a mini browser. It doesn’t have to be fully compliant to be useful. There’s a tons of things you could do with a browser that isn’t spec complaint. In fact, without any JS support at all (which cuts out probably 75% of what a full browser does).</p><p>I hope this little prototype inspires you to go build something amazing.</p><p><a href="https://github.com/joshmarinacci/browser500">Mini Browser repo</a></p>Canvas Computing Prototype 1http://joshondesign.com/2022/12/03/canvas_computinghttp://joshondesign.com/2022/12/03/canvas_computingSat, 03 Dec 2022 06:34:51 +0000<p>I&#x27;ve been working for a while on some new ideas around the future of programming, but I haven&#x27;t done a great job of sharing these with the world. It&#x27;s not science if you don&#x27;t publish your results, so here we go.</p><p>I have long wanted a large infinite canvas for computing. A place where I can sketch, draw, and drop in code, all positioned exactly how I want. I should be able to divide my program into little chunks and algorithms, mixed with interactive controls and visualizers for the output. While the idea seems similar to boxes and lines visual programming environments, mine is still firmly built around a text based language (in this prototype, anyway). I&#x27;m imagining something more like a spreadsheet where you can place your data and your functions wherever you want, tied together with dependencies so any change will automatically recalculate the results.</p><p>Okay, that&#x27;s all pretty abstract. Let&#x27;s discuss a very simple, but concrete, example. </p><p>I want to draw a square. I have a canvas view for the output, and a tiny text editor containing a few lines of code. To specify the color I don&#x27;t want to put it in the code, but instead use a little color picker button. When I change the color the code should automatically execute and update the canvas. This week I built a prototype that can do this:</p><p>Here&#x27;s what it looks like:</p><img src="https://joshondesign.com/images2/infinite_canvas_prototype_v1_closeup.jpg" alt="z"/><p>Each of the panels can be dragged around in an infinite canvas. Not shown is a debugging panel and little menu bar.</p><p>Functionally it works. Everything can be moved around, the code executes, and there are dependencies so that when you change the color the code runs and updates the canvas. Most importantly, the whole deal is persisted as a JSON document into the browser&#x27;s local storage and reloaded on page load.</p><p>Now let&#x27;s discuss what the problems were.</p><h3>Dependency management</h3><p>I built the prototype using React and Typescript. I tried to have a base class for handling events so that the different coding nodes can communicate with each other. As part of this I want the type system to only let you listen for events that the source will actually send. Making this work with inheritance under Typescript is tricky. At one point two versions of what *thought* were the same type were incompatible and tried to insert a &#x27;never&#x27; type. This blog shows a better approach: <a href="https://cgjennings.ca/articles/typescript-events/">Events and listeners in TypeScript (Christopher G. Jennings</a> that I&#x27;ll try next time.</p><p>Next, The ceremony of which node is watching which is complex. It&#x27;s currently done with the aformentioned event system. There has to be a simpler way to express dynamic dependencies. Also, the system itself needs the ability to monitor all of the watching links and show it in the debugger. How can we find out which objects A is listening to, and which event names? So, new design is needed here.</p><h3>Persistence</h3><p>The persistence system is flaky. We need an object model that can be persisted more reliably. Maybe empty constructors. The problem is that the constructors of the classes do computation, but it&#x27;s a lot harder to call those same constructors programmatically when restored from JSON. Instead I&#x27;d like to be able to allocate the object with the right prototype, restore all of the property values, then invoke some sort of &#x27;init&#x27; method to make the object live. Only then will it be shown on screen and start propagating events.</p><p>Another problem with persistence is that the dependencies are not preserved across persistence cycles. And I have to register a fixed list of classes and views to make persistence work correctly. Is there a better way to automate this part so it needs less overhead? Also the debug console and menu shouldn’t be persisted since they aren&#x27;t really a part of the user&#x27;s document.</p><h3>UI improvements</h3><p>Instead of a menubar I&#x27;d like to be able to create an object by right clicking, selecting from a menu, and the new object appears there. This lets me create an object and give it a position at the same time. </p><p>A few other issues:</p><ul><li>The snippet editor needs syntax highlighting.</li><li>The color constant is too big. It needs a different, maybe non-resizable, window view. Still must be draggable, though.</li><li>The persistence engine needs to know when nodes change and when they move, but other nodes only want to know when a dep changes, not when it moves or is resized.</li><li>The system needs a way to add notes &amp; comments and headers &amp; captions.</li></ul><h3>Next steps</h3><p>For the next iteration I want to address the issues above, but with a more complex use case. I&#x27;m considering some floating constants that are for numbers used to draw a fractal tree. Then you can just move some sliders to change the parameters to the fractal.</p>Make Rects Fast, in Rusthttp://joshondesign.com/2022/09/29/make_rects_fast_rusthttp://joshondesign.com/2022/09/29/make_rects_fast_rustThu, 29 Sep 2022 19:47:17 +0000<p>How can you draw a filled rectangle fast? By making it not slow! Great if you are using a GPU to accelerate rectangle drawing for you, but what if you are doing it oldskool with an in memory frame-buffer? You&#x27;d probably write some code like this:</p><pre><code>// assume width is the width of your frame buffer<br/>// assume color is a u32 of ARGB <br/>let ry = rect.y as usize;<br/>let rx = rect.x as usize;<br/>let rh = rect.h as usize;<br/>let rw = rect.w as usize;<br/>for y in 0..rh {<br/> for x in 0..rw {<br/> self.buffer[((ry + y) * width) + rx + x] = color<br/> }<br/>}</code></pre><p>Now, this will work of course, It&#x27;s in fact the simplest thing to do, but if that&#x27;s a lot of calculations we are doing for every pixel even though they are getting the same value. Really we are just copying a 32 bit integer over and over to the same adjecent memory. If this was in C we&#x27;d use memcpy() but we are using Rust and want to be safe, so what can we do?</p><h3>Safely Modifying Vectors in Rust</h3><p>In Rust, you can represent a memory buffer by a Vector of u8 or u32 numbers. Since my project is for an ARGB display, I&#x27;m using <code>Vec&lt;u32&gt;</code>. While you can set values individually with indexes, this is slower than in C because it has to do runtime bounds checking to make sure the you don&#x27;t off the end of the allocated memory. </p><p>For the same reason Rust <em>really</em> doesn&#x27;t want you to copy <em>only parts</em> of one vector to another because that also needs bounds checking. Instead Rust provides functions like <a href="https://doc.rust-lang.org/std/vec/struct.Vec.html#method.copy_from_slice"><em>copy_from_slice</em></a> which let you copy only the entire vector at once, which is fast and only checks bounds once. That&#x27;s fine if we want to manipulate the entire vector, but what if we only want to work with just a part of the vector?</p><p>Vector has a few other interesting methods like <a href="https://doc.rust-lang.org/std/vec/struct.Vec.html#method.split">split</a> and <a href="https://doc.rust-lang.org/std/vec/struct.Vec.html#method.chunks">chunks</a>, which give you access to subsections of the vector as mutable slices. Want to copy into just the half the vector? Split it! Need to chop it up into a bunch of uniform sized chunks, Chunk it!</p><h3>The Plan</h3><p>We are going to compose our solution for drawing rectangles in sections. First we will divide the buffer up into rows for each scan line, and then figure out the part of the rect inside that row. Then we can finally fill it in. Let&#x27;s go.</p><p>First, let&#x27;s calculate how big the buffer is in <code>buffer_bounds</code> and what part of the rectangle will be drawn in <code>fill_bounds. </code>In case the rectangle isn&#x27;t fully inside the buffer we must calculate the intersection first.</p><pre><code>let buffer_bounds = Rect::from_ints(0, 0, width as i32, height as i32);<br/>let fill_bounds = buffer_bounds.intersect(rect);</code></pre><p>Now let&#x27;s create a vector representing a single row of the rectangle. Think of it as a rubber stamp that we can use over and over. Let&#x27;s fill it with the color.</p><pre><code>let mut row = vec![0; fill_bounds.w as usize];<br/>row.fill(color);</code></pre><p>Now we need to access each row of the buffer as a separate slice. We can do this with <code>chunks_exact_mut</code>. All the <code>chunks</code> methods return an iterator over the slices. The <code>exact</code> version ensures we only get chunks of the exact size we want (any remainder will be ignored), and <code>mut</code> means the slices will be mutable. For each <code>row_slice</code> we first check if it intersects with the rectangle we want to fill. If not then just continue.<code> </code></p><pre><code>for (j, row_slice) in self.buffer.chunks_exact_mut(width).enumerate() {<br/> let j = j as i32;<br/> if j &lt; fill_bounds.y {<br/> continue;<br/> }<br/> if j &gt;= fill_bounds.y + fill_bounds.h {<br/> continue;<br/> }</code></pre><p>Now we can split the row of the buffer into three parts, before, after, and the middle part. The middle is the only part we want to draw. We can pull these parts out using <code>split_at_mut</code>. Now we can finally copy from our row to the current slice, just the part we need.</p><pre><code> let (_, after) = row_slice.split_at_mut((fill_bounds.x) as usize);<br/> let (middle, _) = after.split_at_mut((fill_bounds.w) as usize);<br/> middle.copy_from_slice(&amp;row);<br/>}</code></pre><p>That&#x27;s it. Now you can draw a rectangle or other shapes faster than setting each pixel individually. You can see this code in action as part of the next release of IdealOS Clogwench <a href="https://github.com/joshmarinacci/clogwench/blob/master/minibuf/src/lib.rs">here</a>.</p><p></p>Ideal OS Mark 6http://joshondesign.com/2022/09/21/idealos_mark6http://joshondesign.com/2022/09/21/idealos_mark6Wed, 21 Sep 2022 18:43:38 +0000<p>Yeah, it’s been a while, but not forgotten! In between moving homes and jobs and pets I found to do a new release of my IdealOS prototype. And not just a new release, but an actual full rewrite! Let’s take a look at what’s new in <strong>IdealOS Mark 6</strong>!</p><h3>Rewrites</h3><p>The <strong>central</strong> core process is now entirely in <strong>Rust</strong>. No more Javascript! The central core handles message passing, the on-disk database, and access to all actual hardware. This means the central core has to be fasty, secure and reliable. This made it a good fit for Rust.</p><p>The central core now has a <strong>database</strong> built around JSON objects. Apps can create, update, delete, and query these objects over a simple network api. Currently the database is loaded from test data files and changes are not persisted to disk, but this will change in the future. The database powers the <em>Music</em> and <em>People</em> (contacts) apps, and eventually almost everything in the system. The central core now also has an <strong><em>audio service</em></strong><em> which can play MP3s</em> off of disk. This service is implemented with existing Rust libs, since rewriting an MP3 implementation from scratch is way beyond the scope of IdealOS.</p><p>The <strong><em>window manager and mouse/keyboard input</em></strong><strong> </strong>are also in Rust now. While the Mac version still uses SDL2 underneath, the Linux implementation directly uses the <a href="https://docs.kernel.org/fb/api.html">the Linux Frame Buffer API</a> and <a href="https://docs.kernel.org/input/input_kapi.html">mouse &amp; keyboard input APIs</a>. This gives IdealOS the lowest level access to the kernel, which will give us more flexibility and power on the Raspberry Pi in the future.</p><p>On the client side, apps are still in JS, but they are now using a new UI toolkit called <a href="https://github.com/joshmarinacci/thneed-gfx">Thneed-Gfx</a> that actual works, for a change. It’s a very traditional style of tree-based UI toolkit, think <a href="https://en.wikipedia.org/wiki/Swing_(Java)">Swing</a> or <a href="https://developer.apple.com/documentation/uikit">UIKit</a>, but simple to understand and extremely portable. It’s also built in TypeScript so maintenance should be easier. It uses a hacky bitmap font of my own design, and a central theming system that&#x27;s ugly as sin, but it works! It can also run on the web in any HTML Canvas impl, making it useful for other things and easier to test.</p><h3>What does it look like?</h3><p>Here ya go. Runs on Mac in a window or Linux full screen. That photo is on my Raspberry Pi 400 with the official keyboard and mouse.</p><img src="https://joshondesign.com/images2/02_withapps.png" alt="z"/><img src="https://joshondesign.com/images2/03_on_pi400.jpeg" alt="z"/><p>It’s very ugly and very slow. <em>Very slow</em>. That’s because <em>all</em> IPC is local web sockets. When an app wants to draw it sends a drawing command to the central, encoded as JSON, over a web socket. Central parses it, determines where the message should go, re-encodes back to JSON, and sends it to the window manager process, which parses it again, draws to the window’s back buffer, then actually redraws the screen. Whew. That&#x27;s a lot. One day this will all be done lightning fast through shared GPU memory, but for now it’s good enough. <em>Make it work first, then make it right, then make it fast.</em> We’ll get there.</p><p>All the source is on GitHub in the <a href="https://github.com/joshmarinacci/clogwench">Clogwench</a> repo.</p><h3>What’s next?</h3><p>My plan is to continue working from this base. The core is Rust and apps are in Typescript. No more complete rewrites. The next few tasks are:</p><ul><li>fix the central build so you can make and run the whole system with a single command.</li><li>move apps to a separate pure JS repo, also with a single build script.</li><li>heavy work on the <strong>multi-line text editor component</strong>. Editing text is such a fundamental part of an OS that it <em>has</em> to be good.</li><li><strong>improve the database</strong> to actually persist to disk and have higher level semantics like versioning, security, and domains.</li></ul><p>I hope you had a great summer. It’s time to get back to school.</p>Life Moves Fasthttp://joshondesign.com/2022/08/02/live_moves_fasthttp://joshondesign.com/2022/08/02/live_moves_fastTue, 02 Aug 2022 23:19:07 +0000<p>It’s been a while since I’ve talked tech and much longer since I’ve talked about anything personal. There&#x27;s a lot of updates to share, so buckle up.</p><h3>The last two years</h3><p>In mid 2020 I was laid off from Mozilla along with the rest of the research division. I obviously thought this was a poor decision on senior managment&#x27;s part, but that&#x27;s a topic for another day. About 6 months later my wife and I separated and I moved to a nearby apartment where my son lives with me half time. About six months after that my division at Lyft was acquired by Toyota, and then shortly before Christmas of 2021 my mother passed away. </p><p>Needless to say, I&#x27;ve had a lot of change and turmoil in my personal life in the last two years. Fortunately I’ve been working with a good therapist and I have a very supportive family. </p><p>After my mother&#x27;s memorial service in April I realized I need to quit my job and take some time off. I&#x27;ve been on a turbo jet treadmill for the past few years. I need time to really recover and refresh my brain and soul. If I keep going at this pace I&#x27;ll wind up with a nervous breakdown or worse.</p><p>At the end of May I gave my notice at Toyota. While I&#x27;ll miss my co-workers it was definitely the right decision. I&#x27;ve spent the last two months doing things with family, reading for pleasure, and taking a whole lot of naps. It&#x27;s really amazing how fuzzy the human brain can be without enough sleep. </p><p>Right now I&#x27;m focusing on friends and family. With my mom gone my dad really needs me. My niece and nephew need me as well. I don’t know what my future holds but I know that this is what I should be doing right now. I am where I should be. </p><p>Two months probably wasn’t enough to fully heal but it’s been a good start. For the first time in years I’m feeling optimistic about the future. I got to reconnect with family I haven seen I years. I’ve been learning about topics over never had the time to study , like robotics and machine learning. And I’m actually feeling creative again.</p><h3>New Job</h3><p>Yesterday I started a new job. I am now a senior engineering manager at <a href="https://markforged.com">Markforged</a>, the makers of some seriously amazing industrial 3D printers. Below is possibly the strongest <a href="https://www.3dbenchy.com">Benchy</a> ever, printed in nylon reinforced with carbon fiber.</p><img src="https://joshondesign.com/images2/benchy_markforged.jpeg" alt="z"/><p>I’m excited to be working for a non-California company for the first time. Markforged is based in Boston. In fact, I’m visiting there next week. If any of you are in the Boston area let me know. We can grab a drink.</p><p>I’m going to start blogging soon about my robotics and machine learning work, probably with some 3d printing thrown in. I might even start a YouTube channel. The focus will follow much of my previous personal research on how to enhance human intellect and cognition, but this time I’m gonna do it with some different tools it’s gonna be fun.</p><h3>Next Steps</h3><p>First: don’t forget your mental health. Even if you didn’t lose a family member recently, the last two years has been hard on all of us. It’s OK to not be 100%. We really can’t do everything. Take some time for yourself. Relax. Maximum productivity is not the right goal. Nothing matters more than friends and family. Life is made of people.</p><p>I hope you are keeping cool this summer.</p>Why are Browser Engine Monocultures a bad thing?http://joshondesign.com/2022/04/14/browser-engine-monoculturehttp://joshondesign.com/2022/04/14/browser-engine-monocultureThu, 14 Apr 2022 22:33:07 +0000<p>It is often mentioned in Hacker News comments and the Twitters that it’s a tragedy that the web ecosystem is now dependent one only three renderers: Chromium, WebKit, and Gecko. Every time a new browser is announced I see comments like:</p><blockquote>&quot;The world needs less Chrome-clones and Google-backed browsers (which to a certain extent includes Firefox), and more independent ones.&quot; [<a href="https://news.ycombinator.com/item?id=30862374">link</a>]</blockquote><p>and</p><blockquote> &quot;The web really needs a truly cleansheet, non-invasive, open source browser written in a modern language. It’s disheartening to see the consolidation in the web space on Chrome. We need an alternative. There has to be someone who can counter Chrome’s dominance with a modern, fast browser engine. [<a href="https://news.ycombinator.com/item?id=29650963">link</a>]</blockquote><p>I understand this sentiment but I want to challenge it. No non-programmer cares what libraries make up their favorite browser. They don’t care whose JPEG decoder it is inside, or which CSS parser. Why does it make a difference which large browser engine is inside either?</p><p>All three major browser engines are open source. Anyone can see how it works, submit patches, or make a fork. Yes, some companies have undue power over browsers and web standards, but that is due to OS lock-in and market share, not because of the internal rendering library. Adding a fourth browser engine wouldn’t change that.</p><p>So I ask you: </p><p><strong>Why do you care that the browser engine landscape is becoming a mono-culture? Why is a </strong><strong><em>browser engine</em></strong><strong> mono-culture a bad thing? And if the real issue is </strong><strong><em>browser product</em></strong><strong> marketshare dominance, then wouldn’t reusing an existing browser engine be a better choice than using something else?</strong></p>Gameboy Emulator Progresshttp://joshondesign.com/2022/02/23/gb_emu_2http://joshondesign.com/2022/02/23/gb_emu_2Wed, 23 Feb 2022 18:40:39 +0000<p>After a few weeks of work I’ve been able to get Tetris to boot and play. I can also run Dr Mario to enter the play screen but all of the pieces are hidden for some reason.</p><img src="https://joshondesign.com/images2/2022-02-04-tetris-titlescreen.png" alt="z"/><img src="https://joshondesign.com/images2/2022-02-04-drmario-titlescreen.png" alt="z"/><p>I’m not going to walk through all of the code I’ve written (it would be both long and boring) but I do want to touch on a few things I’ve learned, suggestions for using Rust better, and tips that will make the process easier for those who come after me. (full code <a href="https://github.com/joshmarinacci/gb_emu">in the repo</a>)</p><p>Making an emulator is hard because there are so many pieces that must work together perfectly. If one piece is wrong you may see a failure in an entirely different section of the emulator, making it almost impossible to debug. To address this you need three things:, unit tests, test data, and a debugger.</p><h3>Unit Tests</h3><p>I knew building the CPU would be hard, so I started making unit tests. Load up two bytes of memory with an instruction, execute the instruction, check that the registers hold the right values. This worked pretty well for the first few instructions but started to become incredibly repetitive. The GameBoy’s Z80 derived CPU has over <strong>500 op codes</strong>, that’s a lot of unit tests, and many of them are very similar. After ten tests I could tell this wouldn’t scale so I took a different approach: building a DSL using Rust <em>enum</em>s.</p><p>Most instructions are in the pattern of <em>Load, source, destination</em>, or <em>math operation, source, destination</em>. There are many possible combinations, but all of the sources and destinations always a register, an immediate value (meaning it’s in memory right after the instruction), or somewhere in memory pointed to by a value in a register. Knowing this pattern I created a series of Rust enums for the different possible sources, destinations, and operations, which I can assemble into every pattern I need, almost like a DSL (domain specific language). </p><p>Let’s look at an example. Op code <code>6A</code> loads the contents of register <code>D</code> into register <code>C</code>. I started with a <code>match</code> statement of every opcode.</p><pre><code><code>0x006A =&gt; {<br/> let v = cpu.get_reg_D()<br/> cpu.set_reg_L(v)<br/> cpu.pc += 1;<br/>}</code></code></pre><p>This is easy to understand but when the next instruction comes along that is the same, but with register <code>B</code> instead of <code>D</code>, we will start to see a lot of repetitive code. Next I created enums for the different registers so I could load them like this:</p><pre><code><code>0x006A =&gt; {<br/> let v = cpu.get_reg(Reg8::D())<br/> cpu.set_reg(Reg8::L(),v);<br/> cpu.pc += 1;<br/>}</code></code></pre><p>This works but doesn’t really shorten things and doesn’t help me when I come to an instruction that wants an immediate value instead of a register. Eventually I realized that <em>any</em> 8 bit source and destination was interchangeable, so I could use a register enum or one representing an immediate 8 bit value like this:</p><pre><code><code>0x006A =&gt; {<br/> let v = get_source(Source::SrcR8(Imm8())<br/> cpu.set_reg(Dest::DstR8(Reg8:L());<br/> cpu.pc += 1;<br/>}</code></code></pre><p>Since most load operations take the same number of cycles and move the PC (program counter) the same amount, I could condense it all into a DSL like this:</p><pre><code><code>op_table.load8(0x6A, DstR8(L), SrcR8(D));<br/>op_table.load8(0x6B, DstR8(L), SrcR8(E));<br/>op_table.load8(0x16, DstR8(D), Im8());</code></code></pre><p>Each call to <code>load8</code> creates an entry in the op table with all of the information needed to perform each instruction. The actual code is then <strong><em>a match on the enums instead of opcodes</em></strong>, of which there are far fewer, so the code is far smaller. </p><p>The implementation for <strong><em>all load ops</em></strong> now looks like this.</p><pre><code><code>match op {<br/> Load8(dst,src) =&gt; {<br/> let val = src.get_value();<br/> dst.set_value(val);<br/> }<br/> cpu.pc += op.len<br/>}</code></code></pre><p>A side benefit of using enums for everything is that we can add extra methods on the enums for debugging and pretty printing assembly code, which is critical for the next part:</p><h3>You’ll have to write a debugger</h3><p>Yep. It&#x27;s true. As you debug your emulator you’ll find yourself trying to print each step the CPU goes through to find the errors. <em>Load instruction, Load memory, Add, store memory</em>, etc. Over and over again. Eventually it will be too much for <em>println</em> debugging. You’ll have to write <strong>a debugger. Fun!.</strong></p><p>I created an interactive debugger using the <a href="https://docs.rs/console/latest/console/">console</a> Rust crate. It lets you write command line programs that take single key text input. I built a simple program to wait for a keystroke, then execute the next instruction and print out the current registers. If I press certain keystrokes it will execute 16 instructions ahead, or an unlimited number unit it hits the next VBlank (vertical blank). </p><p>As development went along I ended up adding more and more features to the debugger, including dumping VRAM to a PNG so I could see sprites, viewing the current status of the hardware registers, and running until particular interrupts are hit. Without a debugger it would have taken me many times longer to even get Tetris working.</p><img src="https://joshondesign.com/images2/Screen Shot 2022-02-23 at 10.52.21 AM.png" alt="z"/><h3>Test ROMs</h3><p>Once you have a debugger and your basic emulator up and running you’ll want to try running actual gameboy programs. <strong>I do not recommend trying to run a full game like Pokemon or Zelda</strong>. They are too big and you’ll never figure out where it fails. Remember you are emulating a CPU that does hundreds of thousands of operations a second. Full games are too much. Instead <em>start with Tetris or Dr Mario</em> since they were some of the simplest and earliest games written and don’t use memory bank switching or complex interrupts. Even better, try some test ROMs specifically designed for verifying your emulator works.</p><p>I started with <a href="https://github.com/c-sp/gameboy-test-roms">the CPU instruction ROMs by Blargg</a> The main rom requires memory banking, but the individual tests do not. Each test rom will print results to the screen, and also to the serial port, so it’s easy to see where it fails. Even once you get Tetris running these roms are still helpful because they exhaustively test <em>every</em> op code, even the ones that are rarely used.</p><img src="https://joshondesign.com/images2/Screen Shot 2022-02-23 at 10.54.41 AM.png" alt="z"/><p>That’s it for today. Have an 8bit week!</p>Starting a Gameboy Emulatorhttp://joshondesign.com/2022/01/24/gb_emu_01http://joshondesign.com/2022/01/24/gb_emu_01Mon, 24 Jan 2022 17:19:16 +0000<p>A week or so ago I ran across a video called <a href="https://media.ccc.de/v/33c3-8029-the_ultimate_game_boy_talk">The Ultimate Gameboy Talk</a>, and indeed it was. Inspired by the simplicity and elegance of the original Gameboy, I decided to try my hand at building an emulator. A week later I have this:</p><img src="https://joshondesign.com/images2/2022-01-24-hello-github-sdl.png" alt="z"/><h3>Why</h3><p>I&#x27;m fully aware that my obsession with this project is a manifestation of my own mental instability. You see in my day job I work on software, but nothing is in my control. I can&#x27;t control the deadlines. I can&#x27;t control my staff. I have to go with what happens and constantly fight fires. It&#x27;s okay, that&#x27;s why I&#x27;m paid to do my job, but it is no way relaxing. </p><p>Building a gameboy emulator is the exact opposite. The problem is constrained. The hardware is fully documented, there are no more games being made. The specs won&#x27;t change half way through. It&#x27;s sort of like a jigsaw puzzle. I *know* that every piece has a final resting place. That&#x27;s why I find it simulatenously stimulating and relaxing.</p><h3>Rust</h3><p>For this project I&#x27;m using Rust. There already exist several Rust Gameboy emulators, but that&#x27;s not the point. This will force me to learn how to use Rust better. To learn how low level bit twiddling coding is done. Programming languages are tools. The more time you invest in the tool the better you can use it. Since I think Rust is one of the most important languages of the last 20 years, I want to invest a lot.</p><p>Over the next few weeks I&#x27;ll keep adding more until I can at least play Tetris. I&#x27;m a long way from that point. I can barely execute a simple test rom, as you can see above. Even that little bit was an achievement. I had to build over 200 Z80 opcodes and learn about memory access and bank switching just to get to this point. I think I have a pretty good handle on it now, though, and IntelliJ&#x27;s Rust refactoring tools are far better than I expected. Hopefully visual progress will accelerate soon.</p>Roku IDK on Machttp://joshondesign.com/2021/11/20/roku_idk_machttp://joshondesign.com/2021/11/20/roku_idk_macSat, 20 Nov 2021 03:25:35 +0000<p>I have long loved my series of TV streaming Roku devices. The UI isn’t as fancy as the Apple TV, but it’s very stable, very responsive, and far easier to use than Apple’s insane remote. Historically the Roku SDK was really only targeted at streaming TV apps (which makes sense), but there wasn’t a way to write games or other high performance apps using anything but Roku’s own BrightScript SDK. However, about a month ago Roku announced the Independent Developer Kit that lets anyone build and side load apps onto their own Roku’s using C++ and OpenGL. I was thrilled but bummed that you have to run the SDK on Linux. However, it’s just some command line scripts, so maybe we could run it in a Docker container. After a little experimenation I figured out how. Let&#x27;s dig in.</p><h3>Setup Docker</h3><p>First install <a href="https://docs.docker.com/desktop/mac/install/">Docker for Mac</a></p><p>Next, from the terminal on your Mac, create a Linux Ubuntu container image.</p><pre><code><code>docker run -it —name ubuntu ubuntu:xenial bash</code></code></pre><p>Now you should have Ubuntu Linux running inside of a Docker container, and a shell into this container as root. </p><p>From <strong>within the linux container</strong>, install the standard GCC build chain</p><pre><code><code>apt install build-essential</code></code></pre><h3>Setup the IDK</h3><p>Back on your <strong>host macOS terminal</strong>, <a href="https://developer.roku.com/docs/developer-program/idk/idk-getting-started.md">download the IDK</a> and then copy the tarball into your linux container.</p><pre><code><code>docker cp ~/Downloads/10.5.517-IDK.tar ubuntu:/home</code></code></pre><p>back in the <strong>container shell</strong>, uncompress the tarball</p><pre><code><code>tar -xvf 10.5.517-IDK.tar</code></code></pre><p>go to the gles2 example</p><pre><code><code>cd /home/IDK/samples/gles2</code></code></pre><p>and build it</p><pre><code><code>make</code></code></pre><p>Now it will compile the test app and create the file:</p><pre><code><code>idk_sample_gles2.squashfs.bin</code></code></pre><p>Copy this file back to macOS by typing this into a terminal <strong>on the </strong><strong><em>mac side</em></strong></p><pre><code><code>docker cp ubuntu:/home/10.5.517-IDK/samples/gles2/idk_sample_gles2.squashfs.bin .</code></code></pre><p>Now you’ve got a compiled app and you just need to install in on your Roku.</p><h3>Roku Developer Mode</h3><p>Now enable developer mode on your Roku following <a href="https://developer.roku.com/docs/developer-program/getting-started/developer-setup.md#step-1-set-up-your-roku-device-to-enable-developer-settings">these instructions</a>. As part of the setup it will give you a <em>URL</em> on your local network, and <em>username</em>. Write these down. Set a <em>password</em> as part of the setup, and write that down too. The setup process reboot your Roku.</p><p>After the reboot navigate to that <strong>URL</strong> (something like <code>http://10.0.0.121/</code>)in your browser and put in the user/pass. You are now at the app install page. Upload the binary and click install.</p><h3>Enjoy!</h3><p>Done. If you have a supported device the app will launch. Otherwise it will tell you that <em>“IDK Applications are not supported on this device.”</em> I <em>thought</em> my device <a href="https://developer.roku.com/docs/specs/hardware.md#current-roku-models">was supported</a> because it’s under the no longer manufactured but still can update to the latest OS table. However you need to look for the <strong><em>IDK support</em></strong> row, and you’ll see its only some of the current devices. I have a Roku Ultra and there have been several versions with that name but mine doesn’t support the IDK. I suspect it has to do with the particular ARM chip inside, which has clearly changed over time. In any case, I ordered a new <a href="https://www.amazon.com/gp/product/B075XLWML4/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&amp;psc=1">4k streaming stick + (2021 edition) for 30$</a> so in a few days i’ll be able to test my apps.</p><p>I&#x27;m looking forward to messing around with the SDK and building a snow flake simulator for my annual Christmas Countdown clock. Ciao!</p>Make a Markdown Parser with OhmJShttp://joshondesign.com/2021/07/16/ohm_markdown_parserhttp://joshondesign.com/2021/07/16/ohm_markdown_parserFri, 16 Jul 2021 23:40:24 +0000<p>In many of my projects I need to parse something, usually a text based language. Sometimes it&#x27;s inline docs. Sometimes it&#x27;s a custom DSL for generating code. And sometimes it&#x27;s markdown files. My goto tool for this type of work is <a href="https://ohmlang.github.io">OhmJS</a>, an open source parser language for JavaScript based on PEGs. </p><p>OhmJS has a pretty good tutorial and a great online tool for seeing how your parser works, so I won&#x27;t cover that same territory here. Instead we&#x27;re going to build a more complex parser as an example: <strong>Markdown</strong>.</p><p>Markdown is a tricky format because it is not well defined. There are many extensions to it and it has some quirky rules. My first few attempts ended in failure. I could parse the first paragraph but syntax inside the text block would interfere with knowing when the block ended and the next one began. On my third attempt I figured out the trick: <strong>don&#x27;t parse it all at once.</strong> Instead <strong>use two passes</strong>. The first pass finds the block boundaries and the second pass dives inside the blocks. With that crucial decision made the rest is easy. Let&#x27;s take a look.</p><h3>Parsing Blocks</h3><p>Here is the complete parser for the first pass</p><pre><code>MarkdownOuter {<br/> doc = block+<br/> block = blank | h3 | h2 | h1 | bullet | code | para | endline<br/> h3 = &quot;###&quot; rest<br/> h2 = &quot;##&quot; rest<br/> h1 = &quot;#&quot; rest <br/> para = line+ //paragraph is just multiple consecutive lines<br/> bullet = &quot;* &quot; rest (~&quot;*&quot; ~blank rest)*<br/> code = q rest (~q any)* q //anything between the \`\`\` markers<br/> q = &quot;\`\`\`&quot; // start and end code blocks<br/> nl = &quot;\\n&quot; // new line<br/> sp = &quot; &quot;<br/> blank = sp* nl // blank line has only newline<br/> endline = (~nl any)+ end<br/> line = (~nl any)+ nl // line has at least one letter<br/> rest = (~nl any)* nl // everything to the end of the line<br/>}</code></pre><p>The document is defined as a series of blocks. A block can be one of a blank line, the headers, a bullet in a list, a code block, or a generic paragrah. The headers and are paragraph are pretty simple, but the bullet and code need some explanations. </p><p>A code block is delimited with triple backquotes and can contain anything, including newlines and other header markers. To handles this I used the <strong><em>negative lookahead</em></strong><strong> </strong>operator, ~ , which matches everything <em>but</em> the pattern. This way I can say &quot;get anything except the ending <code>```</code>, including newlines&quot;. To keep it simple I defined the triple quotes as <code>q</code> so that I can reuse it in the code pattern: <code>q (~q any)* q.</code> This works perfectly except for one thing: the opening <code>```</code> must be on a line by itself. However, in real world markdown sometimes people put the name of the code snippet&#x27;s language on that line like this:</p><pre><code>``` javascript<br/>console.log(&quot;i&#x27;m some javascript&quot;)<br/>```</code></pre><p>To handle that I defined another rule called <code>rest</code> which slurps up anything until the end of the line. If there is any text there it will be passed to my semantics which can handle it properly instead of trying to parse it here. The final rule is the one you see above with rest in it. Bullet list items are handled by a rule structured the same way as code.</p><h3>Block AST</h3><p>Once the blocks are parsed you can use semantics to turn them into whatever structure you want. In my case I made a tiny AST with the name of the block and it&#x27;s contents as a big string.</p><pre><code>const H1 = (content) =&gt; ({type:&#x27;H1&#x27;, content})<br/>const H2 = (content) =&gt; ({type:&#x27;H2&#x27;,content})<br/>const H3 = (content) =&gt; ({type:&#x27;H3&#x27;,content})<br/>const P = (content) =&gt; ({type:&#x27;P&#x27;,content})<br/>const LI = (content) =&gt; ({type:&#x27;LI&#x27;,content})<br/>const code = (language,content) =&gt; ({type:&#x27;CODE&#x27;, language, content})</code></pre><p>and my semantics look like this:</p><pre><code> parser.semantics.addOperation(&#x27;blocks&#x27;,{<br/> _terminal() { return this.sourceString },<br/> h1:(_,b) =&gt; H1(b.blocks()),<br/> h2:(_,b) =&gt; H2(b.blocks()),<br/> h3:(_,b) =&gt; H3(b.blocks()),<br/> code:(_,name,cod,_2) =&gt; code(name.blocks(),cod.blocks().join(&quot;&quot;)),<br/> para: a=&gt; P(a.sourceString),<br/> blank: (a,b) =&gt; ({type:&#x27;BLANK&#x27;}),<br/> bullet: (a,b,c) =&gt; LI(b.sourceString + c.sourceString),<br/> rest: (a,_) =&gt; a.blocks().join(&quot;&quot;)<br/> })</code></pre><h3>What&#x27;s on the Inside?</h3><p>Now we need to process the contents of the blocks. I do that with a second grammar like this:</p><pre><code>MarkdownInner {<br/> block = para*<br/> para = link | bold | italic | code | plain<br/> plain = ( ~( &quot;*&quot; | &quot;\`&quot; | &quot;[&quot; | &quot;__&quot;) any)+<br/> bold = &quot;*&quot; (~&quot;*&quot; any)* &quot;*&quot;<br/> italic = &quot;__&quot; (~&quot;__&quot; any)* &quot;__&quot;<br/> code = &quot;\`&quot; (~&quot;\`&quot; any)* &quot;\`&quot;<br/> link = &quot;!&quot;? &quot;[&quot; (~&quot;]&quot; any)* &quot;]&quot; &quot;(&quot; (~&quot;)&quot; any)* &quot;)&quot;<br/>}</code></pre><p>This defines a paragraph as a list of spans which can be either links, bold, itlics, inline code, or just plain text. Each sub rule uses a simliar pattern with a negative lookahead. Links use the pattern twice because they have two parts. The operation to process the block contents looks like this.</p><pre><code> parser.semantics.addOperation(&#x27;content&#x27;,{<br/> _terminal() { return this.sourceString },<br/> plain(a) {return [&#x27;plain&#x27;,a.content().join(&quot;&quot;)] },<br/> bold(_1,a,_2) { return [&#x27;bold&#x27;,a.content().join(&quot;&quot;)] },<br/> italic(_1,a,_2) { return [&#x27;italic&#x27;,a.content().join(&quot;&quot;)] },<br/> code:(_1,a,_2) =&gt; [&#x27;code&#x27;,a.content().join(&quot;&quot;)],<br/> link:(img,_1,text,_2,_3,url,_4) =&gt; [&#x27;link&#x27;,<br/> text.content().join(&quot;&quot;),<br/> url.content().join(&quot;&quot;),<br/> img.content().join(&quot;&quot;)]<br/> })</code></pre><p>Agan it&#x27;s using the simple AST for runs of style within the block.</p><p>That&#x27;s pretty much it. The rest of the code attaches the two parsers together with a nice async function. </p><p>I&#x27;ve submitted the full code to the OhmJS repo <a href="https://github.com/joshmarinacci/ohm-1/tree/master/examples/markdown">here</a>.</p>IdealOS Mark 5http://joshondesign.com/2021/07/06/idealos_mark5http://joshondesign.com/2021/07/06/idealos_mark5Tue, 06 Jul 2021 14:08:06 +0000<p>After a few weeks long sprint I’ve got another build ready for Ideal OS. Visually it’s starting to come together nicely. Check it out.</p><img src="https://joshondesign.com/images2/mark5.png" alt="z"/><p>There were tons of little things that went into this build so I&#x27;ll just cover the big stuff.</p><h3>Typescript</h3><p>I refactored the core server to use Typescript and completely rewrote the app / websocket / window management code to be simpler and shorter. A big win for maintainability. I was also able to start rewriting the unit tests in Typescript and make them shorter. Why did I wait so many years before learning typescript!</p><h3>Simple Database</h3><p>I pulled in the database code from one of my other IdealOS experiments from last year. The database is essentially a list of JSON objects that fit particular schemas. Apps can search the database over the wire using queries for things like <em>items in a particular category</em> or <em>fields that match a certain substring</em>. </p><p>As databases go its pretty simple. The magic is that we can bind GUI widgets to queries very easily. Rather than the app doing requests for data and then synchronizing internal state manually, the database itself <strong>becomes that state</strong>. Right now there’s still more glue code than I’d like, but after some polishing you should be able to declare a list of tasks with something like this to handle incremental text search</p><pre><code>let search = new TextBox({text:”abc”})<br/>let list = new ListPanel({<br/> query: AND(<br/> IS<em>TYPE(“note”), <br/> IS</em>SUBSTRING(search.text)<br/> })</code></pre><p>Whenever the user types into the text box the query will be automatically updated. In fact, if the database changes by another process, like a background service fetching new emails, the list panel will automatically include those updates as well. <em>No extra code required!</em></p><p>Once the new database system is polished it will make certain types of apps trivial to build. The DB will do all of the heavy lifting. This is the screenshot of a new “notes” which is entirely database driven as described above.</p><img src="https://joshondesign.com/images2/mark5_notes.png" alt="z"/><h3>Visual Polish</h3><p>I cleaned up the visuals. Mostly lots of tiny things but that collectively add up to something that looks more polished. In particular the light and dark themes are now both fully black and white and represent mirror images of each other. The close and resize controls are using glyphs from the standard font, as are all of the dock icons, icons in the right side of the menubar, and the sidebar widgets.</p><p>Here&#x27;s the dark mode theme.</p><img src="https://joshondesign.com/images2/mark5_dark.png" alt="z"/><h3>New UI components</h3><p>There is now a proper multi-line text box that wraps at the word boundaries, uses cursor bias, and accept actions for all navigation instead of arrow keys. This means in the future moving around in a text box could be done by alternate keyset defined through system wide bindings, rather than apps implementing their own keybindings.</p><p>I also added very ugly list view and scroll panel which required a minimal implementation of clip rects. I’m still debating where clipping should happen, in the app or the display server. It works well enough for now.</p><img src="https://joshondesign.com/images2/mark5_listview.png" alt="z"/><h3>Rust client</h3><p>The other major change is getting the rust client working again. We now have two complete implementations of the display server. One in the web browser in JS and one in rust using real GPU code. The rust client uses SDL2, so in theory it can run on a raspberry pi without having X11 or Wayland running. I haven’t tested this yet, but this lays the groundwork for deploying my stuff as a “real OS” without so much extra scaffolding.</p><p>I’d like to mention something cool I discovered. Everything in ideal os is driven by messages. Most messages are sent to a single receive, like an app. For example, when the user clicks the mouse the display server will send a mouse event to the target application by id. </p><p>Some messages, however, are sent to a category of receivers, not a single one. When an app draws something like a button it sends a drawing event to the display. ‘display’ is a category of receiver. There can be multiple displays registered and they will all get the event. This means we could have multiple display servers running at once and they will all just work.</p><p>Here’s a screenshot of the rust client and web client both running. This isn’t two copies of the OS. There’s only one OS and one instance of each window, just displayed twice by two clients. Since the important state like window positions is kept in the server, the user could click a button in one display and the other will be immediately updated in sync. Pretty cool. </p><img src="https://joshondesign.com/images2/mark5_2displays.png" alt="z"/><p>I hope to use this functionality in the future for other types of clients that must be kept in sync. Maybe a second database that just records all actions into a log file for later analysis. Or maybe multiple keyboards and mice so that two people could use the same computer at the same time?</p><h3>next steps</h3><p>That’s it for this build.</p><p>I didn’t get a chance to mess with audio streams in this sprint, which is probably just as well. There’s simply so much work to do just to get a basic GUI running. </p><p>I still haven’t decided what to work on next. Probably getting the rust client working on a real raspberry pi to prove that this could work without an existing windowing system. And more visual improvements. I’d like to find a way to embed emoji into the text editor using my little 1bit font.</p><p>TTFN - J</p>IdealOS Mark 4http://joshondesign.com/2021/06/18/idealos_mark4http://joshondesign.com/2021/06/18/idealos_mark4Fri, 18 Jun 2021 01:52:11 +0000<p>It&#x27;s been a whole lot of work to get to Mark 4 of Ideal OS. My real goal for this sprint was to have something that at least visually looks like a real operating system. What do you think?</p><img src="https://joshondesign.com/images2/screenshot-2021-06-17-2021.png" alt="z"/><p>Let’s cover the architecture and changes, shall we?</p><h3>Central Server</h3><p>The central server is refactored to have separate manager classes for windows, connections, apps, and finally a separate class for routing messages around. All messages in the entire system go through the primary router which gives us a single point of control. This will come in handy in a minute. </p><p>The server is started with a set of config files which define which apps should run, which apps are available, what services to start, and the available fonts and themes. Because this can be configured unit testing is <em>far</em> easier. I can start the real server with a custom config, stuff in events, and see where it breaks.</p><h3>GUI Toolkit</h3><p>The GUI toolkit is just a library. Technically any app could just look for input events and send out draw calls (many games do this), but in practice most apps will want to use the GUI toolkit.</p><p>The toolkit is a run of the mill tree of widgets with one tree per window. All input and drawing is recursive and, currently, very inefficient. It doesn’t try to track dirty rects or any other optimizations. That can be added later. In the interest of being merely slow and not completely glacial, the entire tree draws to a graphics context which collects all of the draw calls into a buffer and then submits them all at once. No app is <em>required</em> to do this, but it’s an easy optimization.</p><p>Current GUI components in the toolkit:</p><ul><li>button, checkbox, toggle button</li><li>popup selector</li><li>vbox, hbox layouts</li><li>label &amp; multi-line label</li><li>textbox &amp; multi-line textbox (super buggy though)</li><li>icon button (for the dock)</li><li>dropdown menus (for the menubar)</li></ul><img src="https://joshondesign.com/images2/ScreenShot2021-06-17at6.56.46PM.png" alt="z"/><h3>Fonts</h3><p>I moved away from the separate bitmap font and font metrics. Now the standalone font editor loads and saves a single JSON file with both metrics and bitmaps embedded in it. It looks like this:</p><img src="https://joshondesign.com/images2/ScreenShot2021-06-17at6.58.41PM.png" alt="z"/><p>Using the new editor I’ve redrawn the standard ASCII font and used some extra codes for icons and other useful symbols.</p><p>Currently there is just the one font, but the system supports multiple for future use.</p><h3>Visual Themes</h3><p>In addition to handling the usual mouse and keyboard events, the GUI toolkit also supports themes. The server has a CSS like document (currently JSON) which represents the UI theme. For every standard GUI component it lists the colors of the foreground, background, border, etc., including hover and selected states. When a widget draws it requests the values for the current theme. Theme values are cached. If the user changes the current theme an event will be sent to all windows alerting them of the change. This simply clears the cache so that all widgets will fetch new values. </p><p>Currently there are very incomplete light and dark themes. In the future the system will become more advanced, but I’m happy to have the hooks in place now. Apps generally shouldn’t need to care about colors and patterns. They just draw using the current theme values, much like a web page with CSS. Eventually the theme file might become actual CSS. The themes reside entirely in the server. Apps never have to deal directly with the themes.; they just get values from the server. </p><p>The debug app has a button to toggle between the themes and you can see all apps magically refresh themselves.</p><img src="https://joshondesign.com/images2/ScreenShot2021-06-17at6.59.52PM.png" alt="z" data-caption="The dark theme in action"/><p>The dark theme in action. Might be a tad ugly (and buggy!)</p><h3>Keybindings</h3><p>I added the first support for keybindings. Before this release the display manager would send raw keystrokes to the server, which would forward them on directly to the currently focused app. Now there is a keybindings file and an extra step in the server. If a key event matches a keybinding then it will swallow the key event and issue a new action event. This event is for semantically meaningful things that should be common across apps.</p><p>Currently there are only keybindings for up,down,left,right navigation, but over time we will extend it to more. The magic part about this is that apps are completely unaware of the process. They just listen for common action events and don’t have to care what particularly keybindings the user has set. At some point we could even support non-keyboard events. Maybe a mouse gesture could trigger the volume up event, or an assistive device could navigate using breath controls. The point is that apps don’t have to be written to specifically support these devices. It will all be a part of the keybinding system.</p><h3>Translations</h3><p>Common text that is language specific can now be stored in a translations file. Any widget like buttons can be configured with a <code>text_key</code> instead of <code>text</code>. This will request the translation from the server before rendering (cached for performance). As with themes, if the global language changes all apps receive an event, clear their caches, and repaint. </p><p>Currently there is just english and lolcat, with only a few translations. This will develop more in the future.</p><img src="https://joshondesign.com/images2/ScreenShot2021-06-17at7.01.32PM.png" alt="z" data-caption="some controls switched to lolcat"/><p>Some controls translated to lolcat.</p><h3>Display Manager</h3><p>The display manager is just another app which registers itself as a screen, then all graphics calls are routed to it. It also has direct access to the mouse and keyboard and uses them to send out events. </p><p>Though the display manager is also a window manager and can allocate windows and set which is focused, it is <em>not</em> the single source of truth. The server has an authoritative list of apps and windows. This means if the display manager crashes it can restart, reconnect to the server, and fetch all essential state. This makes the whole system more resilient. It also makes development easier because I can be actively developing and restarting the display manage constantly and the other apps are completely unaware. In the future we could even have multiple display managers active at once, with the apps being completely unaware.</p><h3>Windows</h3><p>There are now multiple classes of windows. Standard windows are created with the type <code>plain</code> and the display manager will draw them with title bars, close buttons, and resize areas. Now there are a few other window types:</p><ul><li><em>popup</em>: This is a child window of a plain window. It will be drawn without chrome and is forced to disappear when it loses focus.</li><li><em>menubar</em>, <em>dock</em>, <em>sidebar</em>: these are specialty types that are drawn without titlebars and other decorations.</li><li><em>widget</em>: this is a window which is not draw directly to the screen. Instead it is considered part of another app. Whenever a widget window draws its events will be routed to its owner window, which can then modify the draw calls before sending it on to the real display. The same happens for input events. Essentially it lets one app embed another one inside of it. The is currently used only for the sidebar widgets which are embedded inside of the sidebar app. However it is a powerful concept that I hope to use more throughout the system in the future</li></ul><h3>Apps</h3><p>All apps are separate nodeJS processes that can only communicate through the single message bus. All input, output, drawing, and everything else goes through this single channel, which enforces security. Of course they are currently full node apps and <em>could</em> mess with the system through native APIs, but eventually they will be isolated in containers. </p><p>I’ve added a few more apps, so now the full list is:</p><ul><li><em>fractal</em>: drawing a Mandelbrot set</li><li><em>debug</em>: toggles for theme and language</li><li><em>guitest</em>: shows all of the currently supported widgets</li><li><em>jessecalc</em>: a simple calculator</li><li><em>sidebar widgets</em> for weather, time, and music (w/ dummy data)</li><li><em>texteditor</em>: a simple text editor</li><li><em>todolist</em>: a list of three todos with functioning checkboxes</li></ul><p>The sidebar, dock, and menubar are also separate apps, though with special capabilities because they are part of the system and not 3rd party apps. The menubar receives <code>set_main_menu</code> events, the dock is allowed to launch apps, and the sidebar uses window embedding for its widgets.</p><h3>Next Steps</h3><p>Now that the system basically works for running simple apps with windows I think it’s time to make the server more robust and start building new display managers. </p><p>The only display server today is called Sidecar. (actually that’s not technically true. There is another headless display manager used for automated testing, but it doesn’t actually draw anything).</p><img src="https://joshondesign.com/images2/ScreenShot2021-06-17at7.03.28PM.png" alt="z" data-caption="sidecar tool"/><p>This is the SideCar tool in action</p><p>Sidecar is a web-based testing app that handles drawing and mouse interactions, plus some graphical overlays for determining which windows are part of which apps, where the cursor is, and monitoring logs of the whole system. In short it’s a debugging tool. While it’s great for that, it’s slow and requires a whole web browser to run.</p><p>Next I want to build a second display manager in Rust using hw accelerated graphics. This will accomplish two goals. First it will flush out bugs by having a second implementation of the protocol. Next it will get us closer to being able to run on a Raspberry Pi without an existing window system (X11 or Wayland). </p><p>I also want to rewrite the server backend to be cleaner and more type safe. I’m currently looking at using typescript to make the code safer while still letting me use the flexible Node JS ecosystem. At some point I might rewrite the central server in Rust, but I think that’s a ways off.</p><p>Oh, and I want to have a real audio subsystem. Much as it’s interesting to be able to route event sand drawing command around, I’d like to be able to stream audio between apps and various inputs and outputs. I’m currently looking at some nodes server side WebAudio implementations to see if it does what I need. Right now it can</p><p>The final big feature is to fully integrate a database. This has always been a core concept of IdealOS. I’ve prototyped db driven apps using React in a different repo. I think it’s time to bring this into the core and have some full end to end computation.</p><p>So for next time we’re looking at:</p><ul><li>New display manager in Rust</li><li>Refactor central server with Typescript</li><li>Research audio streams</li><li>Integrate database</li><li>more GUI components (list, scroll box, table)</li><li>tons of bug fixes</li><li>some apps that really read and write data to the database</li></ul><p>try it out</p><p>If you want to try IdealOS yourself you’ll need to check out <a href="https://github.com/joshmarinacci/idealos_bottom">these</a> <a href="https://github.com/joshmarinacci/idealos_sidecar">two</a> repos.<br/>In the <a href="https://github.com/joshmarinacci/idealos_bottom">first repo</a> do <code>npm install; npm run server</code> which will start the central server and some apps.<br/>In the <a href="https://github.com/joshmarinacci/idealos_sidecar">second repo</a> do <code>npm install; npm run start</code> which will launch Sidecar in your browser and connect to the central server.</p><p>If you run across any problems <a href="https://twitter.com/joshmarinacci/">tweet me</a>.</p>IdealOS Mark 3http://joshondesign.com/2021/05/27/idealos_mark3http://joshondesign.com/2021/05/27/idealos_mark3Thu, 27 May 2021 02:00:01 +0000<p>As I mentioned <a href="https://joshondesign.com/2021/04/15/emu_pi_mac">before</a> I’ve gone back to working on the bottom half of Ideal OS. So far I’ve got a messaging protocol, a central server, a few tiny apps, and three different display server implementations. The version I’m calling Mark 3 looks like this:</p><img src="https://joshondesign.com/images2/mark3.png" alt="z"/><p>It has a somewhat working menubar, GUI toolkit with buttons, text input, and labels, a fractal generator, and a little tiny dock to launch apps. More importantly the window manager is fully abstracted from the apps. Apps can draw into a window surface through drawing commands, but they have no control over where the window is or if it’s even visible on the screen. The display server (or it’s delegate) is entirely responsible for drawing to the real screen, positioning and dragging windows, and who gets the input focus. Essentially all apps are untrusted, and must be prevented from interefering with the rest of the system.</p><p>This trust nothing model is probably closer to a mobile OS than traditional desktop ones, but I think it’s a better fit for the modern reality of apps that could come from anywhere. It also encourages pulling features out of the apps and into the trusted system. For example, with a built-in system wide database most apps will never need filesystem access at all. They can read and write safely to the database. Most apps don’t need raw keystrokes, they can accept action events form the system, events which <em>might</em> come from a real keyboard, or a scripting system, or my son’s crazy Arduino creations. Hiding that from the apps makes them more flexible.</p><p>But enough about ideas. Let’s talk about what actually works.</p><h3>Messages</h3><p>Everything is built around typed messages (using code generated from a message schema) sent through out the system over web sockets. It’s horribly inefficient, but incredibly flexible and works across any language or platform. I’ll eventually add more efficient protocols but I plan to always keep web sockets as a backup for maximum flexibility.</p><h3>Keyboard</h3><p>Keyboard: I’ve rewritten the keyboard events twice and probably will a few times more. For now I’ve settled on something that resembles the most recent <a href="link">DOM keyboard event spec</a>. It’s not idea, but it works well enough for a simple text box.</p><h3>Fonts</h3><p>Fonts: fonts are a hard problem. They involve string parsing, graphics apis, anti-aliasing, and the layout system. If there’s one thing that connects almost everything in the system together, it’s fonts. The whole stack has to be functional for fonts to work right. Since I don’t have a whole functional stack yet, I cheated. I drew some bitmap fonts by hand with my own tiny pixel based font format as JSON files. This works well enough to layout a single line of text at one font size and weight, ie: enough to get the rest of the system running. Obviously fonts are something I’ll revisit later once the graphics system improves.</p><h3>Graphics</h3><p>Speaking of which, Graphics! Right now all drawing is done with draw calls from apps to the display server, referencing an x/y point on the apps’ own target windows. Only draw rect, draw pixel, and draw image is supported, but it’s enough to get things working. The display server backs each bitmap with a buffer for fast compositing, but it doesn’t have to work this way. Each app <em>could</em> just draw directly to the screen if the display server wants to do it that way. The API is immediate mode, not retained. If the display server isn’t retaining buffers it can just ask apps to redraw their whole screens. Not efficient of course, but it works. </p><p>Text was originally draw pixel by pixel, but now send bitmaps over the wire. In the future bitmaps could be stored as reusable resources, even aggregated into GPU textures. I’m always trying to enable what will work now, while not preventing future optimizations.</p><h3>GUI toolkit</h3><p>There are many ways to write a GUI toolkit with buttons and lists and scrolling windows and the like. I don’t have time to write a good one right now, so I’ve made the simplest thing that could possibly work, with no optimizations. Each window has a tree of components and containers. Updating has three phases.</p><ol><li>First keyboard and mouse events from the display server. The window sends these events to the input() method on the correct component in the tree. Each component uses the input to update it’s internal state, then call repaint on its’ parent. Eventually this goes up to the window.</li><li>The window then recursively calls layout() on the component tree. Each child can set its’ own size, but its’ parent can override that size if it chooses. For example, a vertical box will size each of its’ children, then position them vertically. All components are laid out on every pass, there is not optimization or caching.</li><li>Once the layout pass has completed the window recursively calls draw(). This makes each component actually generate the draw calls which are immediately sent to the display server. Currently the display server exhibits flickering because calls are sent continually. In the future these could be batched up or pruned for better performance and no flickering.</li></ol><h3>Windows</h3><p>The windows themselves are trickier than GUI components. A component only exists within the app and is used to generate draw calls. The window structure, on the other hand, has to exist in the app, in the display server, and in the main routing server. Further more they must all be kept in sync without letting the app violate any of it’s security restrictions. This is proving trick.</p><p>My current strategy is that the main server is the single source of truth. It keeps a list of all apps and their windows and their child windows. Deciding how to size and position windows is left up to the display server, since it has a better idea of what is a good layout. However, the state is still kept in the main server in case the display server crashes or there are multiple outputs. Finally, events for window changes are sent to the app, but the app has no control over the window position or size. It can request certain values but it ultimately takes what the rest of the system decides. This is especially important for things like popup windows where the position really requires knowing what else is on the screen, which of course is invisible to a non-trusted app.</p><p>I hope this window abstraction system will enable new window features like splitting them across screens or embedding them securely inside of other apps. What if the widgets in a sidebar were just tiny windows like any other, no special apis needed!</p><h3>What’s next?</h3><p>The general philosophy here is using as few abstractions as possible with maximum flexibility. It’s the only way I’ll be able to build a whole system by myself in less than a few lifetimes. I’m not happy with current custom schema system for messages. It’s very clunky and hard to maintain. I plan to switch to JSON schema or similar as it will let me make the messages strongly typed as well as let all system resources be easily editable JSON files instead of custom formats. </p><p>Once schemas are integrated I’m going to move all app, theme, keybindings, and translations into JSON files that are easily loaded and searched with a real query language instead of so much custom code.</p><p>Onward and upward!</p>