wykwit

LLM subscriptions

2026-04-17T00:00:00+00:00

Another post in my LLM series, this time about LLM inference providers, labs, available subscriptions, and why it might be a worthwhile purchase at the time of writing. I also sprinkle in some LLM updates that you might have missed and round it out with my own perspective.

Inference and how to get it

Inference is the prediction made by the model as to what (output) tokens should follow other (input) tokens under consideration. In simple terms, by LLM inference we mean computations delegated to the model that give some output tokens in return. This inference computation is not cheap and usually constrained by the provider (the company or datacenter that runs the LLM).

There are a few ways to obtain it:

using built-in integrations (like the G00gle Search "AI Overview")
using dedicated tools (like Perplexity or DeepSeek apps)
buying a subscription from the model provider
accessing models directly through API calls to the model provider

As we go from top to bottom these options get more expensive, but they do get more flexible too.

Integrations and products

With a built-in integration you have no control over what models are in use, from what provider, what are the model capabilities. You are handed a ready-made product, sometimes only a fancy name, and expected to use it as intended. Search summary is a good example of such integration done right - many people started to rely on it and for certain queries that is more than you need. Other times these "AI" features land anywhere between an annoyance and an obstacle that the user has to fight in order to achieve the result they need. One of many reasons why "AI" gets well-deserved bad rep.

The situation is a little better with dedicated tools. Usually the choice of a tool is either tied to the provider in the first place or allows for switching the underlying model and selecting some preferences. Here's a very good example: T3 Chat is an LLM chat webapp that gives you access to all models for a monthly subscription fee. You get to choose which model you want to use, but the scope of this product subscription is limited to the product itself. You are not buying a subscription for the model inference, you are buying a subscription to access the product. The inference is just a part of it, required to make the product work. Another example that comes to mind is Perplexity. They charge a monthly fee for LLM powered websearch, giving you the option to choose which exact model you want to use for generating the answers. They only support a few selected models. The subscription cost is mainly for access to their app, not meant to provide you with raw model inference.

The cost of inference

Buying inference directly can be rather costly depending on the model. The best way to explore this option is through OpenRouter which is a common gateway for accessing different models served by different providers. You can clearly see how individual providers differ with the service quality: different latency, different uptime statistics, reliability, and most importantly - different inference throughput (tokens per second) for the same model. If you're building a product on top of LLMs, this is the way to go. You pay exactly for what you use. Picking the right model and provider will make all the difference for your project.

For personal use subscriptions may be the best deal right now. If you're not using LLMs much it will probably cost you more than paying for inference directly, but then again if you're not using LLMs much you may as well rely on the other two options discussed previously and avoid any additional costs. There are still many people who keep their subscriptions even though they get nowhere near the usage they pay for. That's one of the reasons why inference subscriptions can stay so cheap. They are heavily subsidized and many companies see them as an opportunity to sell people another product that isn't just inference. As long as you don't fall for this marketing trap, for heavy LLM usage, this comes out to be a really good deal. There are many subscriptions that live on the edge between being a product price and inference price. You should know what you're paying for and keep your hand on the pulse to drop subscriptions that lose their value. The market now is rather competitive, so there's plenty of options to choose from.

Best deals last year

GitHub Copilot

Last year my favorite option was the GitHub Copilot Pro plan. It's priced at 10 USD and gives you a monthly quota of "300 premium request tokens" which can be used for calling top models from 0penAI, G00gle, and Anthr0pic. The way it used to work was that it would consume a predefined amount of "premium request tokens" for each started session, so you could pretty much use a model indefinitely for free after that. Now each message consumes some predefined amount of premium request tokens (probably as intended from the start). Still, this amount is defined per model, regardless of the selected reasoning effort. For a single run of GPT-5.4 xhigh, which can take hours to finish and burn a ton of inference tokens, it is still a decent price. Obviously it is worse than what we had last year. Back then it was probably the most cost-effective option of all. It wasn't as straightforward to use in external tools as it is now, but it was always possible. Between the "premium request tokens" consumption fix and the release of GPT-5.4 that subscription was not worth the price. You could burn through the monthly quota in a matter of minutes trying to use it with Claude Opus.

Perplexity

There was also a brief moment in September last year when you could get a free Perplexity Pro annual subscription. I got on board and I'm getting quite a lot of use from it, often defaulting to the Perplexity mobile app for search. I kinda like it, but to be honest I would never use it if I had to pay for it at all. I set a reminder when I signed up to cancel and delete my account before this promotional subscription runs out. I don't even know how much exactly it costs now, but whatever it is I wouldn't say it's worth the money.

Z.AI

On a different note, another great catch of insane value from last year was the Z.AI GLM Coding Plan. I fell in love with the GLM-4.7 model when I finally tried it in December and I've been preaching it since.

The Lite plan, which is now 18 USD/month, was 3 USD/month back then. I got it, I tried it out, and I signed up for the annual subscription of the Pro plan. If I recall correctly I paid something like 180 USD for it. Now it's priced at 864 USD annually or 72 USD/month. I kinda wish I got the Max plan back then, because it was only slightly more expensive, but I doubt I could push it to the limit. Even on the Lite plan hitting a limit was very difficult.

GLM-4.7 is still a wonderful model and I knew I'd be very happy with it even if Z.AI didn't deliver another one for a year, but they followed with even stronger GLM-5 in February, then GLM-5-Turbo, and then recently GLM-5.1 - all extremely good and very underrated. In benchmarks GLM models are second only to GPT-5.4 and Opus 4.6 with high reasoning effort. In practice GLM models are often way better, depending on the task.

It's also worth noting how much cheaper GLM models are, how much faster, and how much more transparent. They are open-weight, MIT-licensed! The quality of GPT and Claude may degrade over time, we may lose access to these models eventually, but GLM models will stay available, you can download them right now yourself. They are only getting better with time as providers figure out how to run the model more efficiently at higher throughput. That gives me so much more confidence to rely on it, as opposed to constrained private western models.

Long story short - Z.AI's GLM models are my favorite and I'm really glad I got in at the right time.

Fierce competition in China

There are many strong competitors coming from China other than Z.AI. The main one is MiniMax, a company with a very wide offer of generative tools. They timed their previous two LLM releases to match up with Z.AI: M2.1 released a single day after GLM-4.7, M2.5 released the same day as GLM-5. About a month ago they released a new M2.7. I couldn't find a strong enough use-case for the previous two models, so I wasn't particularly keeping up with them, and I haven't yet tried M2.7. They have their own subscription offer. It's hard for me to assess the value, since I am not using it myself, but it does look promising.

I have, however, used a lot of the very impressive Kimi K2.5 model after it released in late January this year. The Moonshot AI lab is a world leader when it comes to LLM research. For some time between GLM releases their model was leading in the intelligence benchmarks too. Moonshot AI offers subscription plans through Kimi Code membership. The prices start at 20 USD/month. Back when I tried their lowest subscription tier I'd burn through the quota rather fast. I heard they have increased it since, but I'm not sure how much you can really get out of these plans. There are stronger models out there and better priced subscriptions. Kimi K2.5 itself is rather cheap even when you pay directly for the inference per token. I do recommend trying out the model, but not necessarily through this subscription. Definitely keep an eye out on future releases and the research that this team drops.

When it comes to research teams the uncontested king is definitely DeepSeek. They have no subscriptions. They don't even charge for the use of their app. The LLM itself is also crazy cheap and very good. Everybody is waiting for the next LLM drop from this lab, because you can bet it's gonna be big. Right now their latest model is the weakest on my list, but this will very likely change with their next highly anticipated release.

You can't talk about LLM research without mentioning Qwen, a research team from Alibaba. Qwen models act as a de facto reference point in LLM research. It's a shame the team didn't receive enough support within their company and their leader had to step down. Fingers crossed they can start up something even better independently. Alibaba does have a Coding Plan subscription, but it seems rather limited and priced at 50 USD/month, so in my eyes not worth considering seriously.

Xiaomi entered the race not too long ago with their impressive MiMo V2 Pro model. On benchmarks it lands very close to MiniMax M2.7, which is very good. Xiaomi also introduced a compelling offer of MiMo Token Plan subscriptions this month, clearly inspired by the Z.AI GLM Coding Plan offer from December. Even website colors are similar. The quota seems very generous judging by their claims. I'm sure MiMo models will only improve with time too, so this may turn out to be one of those rare hidden gems. It can go both ways. Limits on these plans can change any time too, so it's not a given that an interesting offer now will stay reliably good a few weeks or months later.

To wrap up this section, here's how a leaderboard of these LLMs would look according to benchmarks:

GLM-5.1 (Z.AI)
Qwen 3.6 Plus
MiniMax M2.7
MiMo V2 Pro (Xiaomi)
Kimi K2.5 (Moonshot AI)
DeepSeek V3.2

The American offer

OpenCode became an inference provider too with their pay-as-you-go service called OpenCode Zen. They have also experimented with a subscription model through limited-availability OpenCode Black. They've now introduced OpenCode Go, which is a cheaper 10 USD/month subscription that enables access to the open-weight models we've discussed above. There are usage limits in place, but the offer seems more than fair. If you were to pick only one cheap subscription out of all, this should probably be the one.

All the subscriptions discussed so far can be used directly in OpenCode. Major model labs from the land of the free do not give you such liberty. 0penAI GPT, Anthr0pic Claude, and G00gle Gem1n1 all offer similar types of subscriptions, but only 0penAI encourages people to use it however they want. With the other two it's all about working around bullshit restrictions to get any meaningful use of it. Reportedly for both GPT and Claude you may get up to 20x worth of tokens for the price.

I bought ChatGPT Plus for 20 USD/month only to get access to GPT-5.3-Codex on the release day and I've kept the subscription since. I use it only for accessing GPT-5.4 Fast in OpenCode and pretty much nothing else at all. Other perks and features of this subscription are completely useless. Still, I feel like I burn through a lot of tokens with GPT-5.4 and no other subscription could really satiate that. It has been the best model available for a while now - especially useful when fixing things or debugging. For quick implementation work or targeted changes I much prefer GLM models, but I still find GPT to complement it pretty well. If another model comes along I might drop this subscription. I don't see why anybody would pay that much for ChatGPT, but if you want to get some of those sweet subsidized GPT tokens in your tools, then it is a pretty good purchase. From the offers of the three major labs discussed here I'd pick this as number one. Also, Codex TUI is probably the second best tool after OpenCode.

Claude models are infamous for how expensive they are. Even with a ton of subsidized tokens, those are very expensive tokens. I don't think they really carry the value that Anthr0pic claims. That's why I'm skeptical about Claude subscription plans. They are overpriced on two levels. Sure, you get a little bit more Claude Opus use, but if you want the limit to last you for the whole programming session you have to use Claude Sonnet most of the time anyway - and that model is way worse than GLM. I admit Claude Opus can be fun sometimes, but in my opinion Claude models overall are simply overrated. Sure they follow instructions well, but the amount of instructions people feed them to get anything half-decent out is insane to me. I can see why people who over-invest in those complex instructions, MCP configurations, custom skills, and other bloat - why they would prefer a model like Claude. For me this is rarely a top choice when I pick an LLM for the task. Anthr0pic has a bad vibe. Also, Claude Code is probably the worst TUI program ever and Anthr0pic really wants you to use that and nothing else. If you like fighting your tools and have the money to spend this might be a good option for you.

There is a way to make Claude subscriptions work with OpenCode anyway. You need two things: 1. opencode-with-claude plugin for OpenCode and 2. meridian bridge that uses the blessed Claude Code SDK to serve models through a standard API. It's a little convoluted, but the end result is a very smooth experience. Definitely worth setting up to avoid dealing with the horrible Claude Code and Claude-specific configurations.

It's easy to forget that G00gle also has G00gle "AI" subscription plans with Gem1n1. They also offer Claude models through Antigravity IDE, the descendant of Windsurf and another VSCode fork. Apparently these subscriptions can be bundled with other enterprise purchases, so there's a chance your organization already has access to higher quotas with Antigravity. Unfortunately, it's the same story as with Gemini-CLI - their tools are not very good. You may plug this subscription into OpenCode through a custom plugin like opencode-antigravity-auth, but honestly it's not a good experience. This only makes sense if you want to milk some more free inference with the access you already have. If you really want to try out Gem1n1 and Claude along the GPT models I believe you'd be better off paying for a GitHub Copilot plan or Curs0r instead.

Curs0r exists too with their monthly subscriptions, but it doesn't exactly appeal to me. I already have my code editor. I don't use inline auto-complete either. The Cursor Agent is fine, but for the price it's not much. The good thing is you can access a variety of models through their service, including their exclusive Composer 2 based on Kimi K2.5.

Let it all burn

Mostly around the beginning of the year people were infatuated with the idea of agent orchestration. LLMs can write most of the code. Why can't they also work autonomously on the programming projects? Well, we tried it and failed. Notably Anthr0pic failed to build a C compiler and Curs0r failed to build a browser. They went all the way and repainted their experiments as a huge success afterwards. It's quite obviously the opposite, but this kind of trick "can have a strong influence on the weak-minded".

Two other independent attempts ended up as crypto-scams: Geoffrey Huntley's Ralph Loop, and Steve Yegge's Gas Town. Those people are truly the worst charlatans of the AI revolution. It was obvious from all their posts that made the rounds on Hacker News, with claims impossible to reproduce. Unfortunately, the Valley seems to love loud characters like that and somehow they get away with it in the end. At least now there's undeniable proof to show they were full of shit from the very beginning.

So how does one genuinely burn through all these tokens? Turns out good human guidance is crucial for all of this agentic stuff to work well. That's why good tools focus on expanding the human ability instead of replacing it. Two great recent examples: OpenClaw and T3 Code. An LLM-assisted loop of planning, implementing, and reviewing wastes inference like crazy, but yields good results. Preferably involve different LLMs in each step for a little variance. Using custom commands is a good way to make this process more streamlined.

Subscriptions summary

Here's a list of discussed subscriptions and how I would group them in terms of value at current prices.

Most subsidized:

0penAI ChatGPT - 20/100/200 USD
Anthr0pic Claude - 20/100/200 USD
MiMo Token Plan - 6/16/50/100 USD

Strong contenders:

MiniMax Token Plan - 10/20/40/50/80/150 USD
Z.AI GLM Coding Plan - 18/72/160 USD
Kimi Code - 20/40/100/200 USD

Most flexible:

OpenCode Go - 10 USD
GitHub Copilot - 10/40 USD
Curs0r - 20/60/200 USD

What do I use?

Personally I mostly make use of my Z.AI Coding Plan which is generous enough that I never really run out of inference for my favorite GLM models. I also keep a ChatGPT Plus subscription for certain things - GPT-5.4 is very capable and useful, but I'm riding the limit almost all the time. American models quickly lead you astray; in personal projects it's not as fun. If I were to buy another personal subscription, I would go with OpenCode Go. Every month I also squeeze out whatever little inference I can from the employer-provided Curs0r and GitHub Copilot subscriptions. Unlike personal subscriptions, these run out fast.

I burn through most of this inference using OpenCode (mainly in a TUI-first workflow, but sometimes using the WebUI as well), running multiple sessions in git worktrees managed with my tool bra, and monitoring each project and branch directory with the help of my madgit. I've also built out a dedicated web dashboard to monitor all my subscription quotas: llm-usage-monitor.

As you'd probably expect - I mostly rely on open-source software, including a lot of my own. Similarly, I prefer to rely on open-weight models, even if not self-hosted. It would be foolish not to bend this a little to make good use of the deals around, but still for me the core must stay open, and not just "open" in name only.

Blog evolution

2025-11-02T00:00:00+00:00

For years I've been rocking the same minimal design on my blog. I've been creating web content of some sort since I first learnt how to make basic websites back in primary school. I feel like the form of my personal website became inadequate with passing time and as my content continues evolving. It was time for a little make over. Most importantly, I had to improve meta info for my posts and unify the style of all pages. I've decided to focus on functional improvements rather than bold visual changes.

My little blogging evolution

When I first started this blog I had a different goal in mind than today. I wanted to make my dense notes public on the web so that I could refer to them whenever I needed. Let's take "Static IP" blog post as an example. There is almost no introduction, the post starts with a link to relevant Arch Wiki article, then there's a few command examples with short descriptions, and that's it. If I ever forget how to use the 'ip' command (which regularly happens) I can simply go back to that very post and the list of all the commands I need is right there. My blog was a collection of notes. Titles were as short as possible, I had no need for subtitles or catchy headers, only enough information for me to find what I knew I had saved. What little design there was served that purpose very well. However, it was not very approachable for others.

Before 2018 my blog was more of a little personal journal and that was... cringy. Some things are better left off the internet. Now I write my personal notes on paper, those can be ephemeral. I publish more deliberately, being concious about the fact that things here may stay alive longer and are generally available to see for everyone with no restrictions. I always try to keep some hard technical info in my posts. I still use the blog to note down things I would like to refer to in the future, but my old style of strict notes does not work with the things I learn anymore. I want to share more things I find particularly interesting and use this platform for more than just stashing notes. With the shift to more opinionated articles the form does not serve the function well enough anymore. I once again find the need for subsections, descriptions, and meta info for each page on the blog.

I published my blog on neocities over 5 years ago. Neocities websites were always my inspiration, and especially for my "about me" page back then. I really liked the way it turned out - minimal, clean and informative. However, it did not fit in stylistically with the rest. Regardless, I liked it so much that it just had to stay.

One of the redesign goals was to unify the style across all pages on this website: blog (article list), post (article content), network (links list), and the "about me" page. I went with the minimalist dark look from my blog, and adapted the "about me" page to better fit in with that style.

Design principles

Core design principles did not change. I need my website to be readable without JS, accessible from a terminal browser, very hacker-friendly. I want to use a dark theme by default. There should be no unnecessary distractions on the page. Content is the focus. If there is JS, it's only there to help the user, not as a requirement to view the website.

Frontend was never my strongest suite, but recently I came around to using modern hybrid web frameworks and I completely fell in love with Nuxt, which is built on top of Vue.js. I've been debating whether to use Nuxt or a different SSG for this website. I'm sure I want to use Nuxt for other websites in the future, but for this one a simpler SSG like Zola seemed to be a better fit. Until now I've been using JS to fetch the post content on the blog page, with links to text-only versions of the post in case the user does not support JS. I would re-generate JSON with a list of all posts on every publish with a little python script in my website repo. I could just as well use some SSG by hand or regenerate the website in a continous delivery pipeline on GitLab when I push to master. Maintainability should not really be a concern with statically generated pages.

I considered including some Visual Sugar: animations, page fade-in, code syntax highlighting. There are no long animations, so users don't have to wait for the content to display for too many millisecons. Adding syntax higlighting for code snippets was long overdue.

I was also debating whether or not to include more links on the home page, to things such as: recent posts, recently updated projects, maybe fediverse activity. Right now the home page is a simple landing page hub from which you can go to my blog or the "about me" page, and only from there do you get to posts and other places. The landing page gets many accidental visits and bot traffic, so I prefer to have the blog at least one click away from the home page. On one hand I like how minimal it is, but on the oter hand - it makes sense to better display what I'm doing here. As of now I haven't made that move yet.

Finally - metrics. I changed my mind about this, because for years I was strongly against it. I'd never have any tracking on the website, not even to count the page view. This time I would prefer to know which pages and posts attract most eyeballs and how many people are even reading this. For this I decided to use Umami analytics - it's very simple, self-hosted, open-source, privacy-respecting, no-cookie-bullshit, presumably GDPR and PECR compliant, easy to block if you care, and at the same time should give me enough insight for all I need.

Eventually I might also include a comments section for each article. If I built out a little dynamic service to gather user feedback, it would bring even more valuable insight, and make the website that much more alive. Still, I don't want to make any promises and I need to think it through a little more.

Results

When I first approached the idea of a redesign my imagination was running wild with all the things I could do. It was on my mind for over two months. The scope was so broad that no work was ever going to get done.

Only when I embraced small, focused and purposeful changes, did I make any progress in this effort. Instead of re-building from nothing, I took what I already had and improved it.

For now I'm pretty happy with how this website presents itself online. Migrating to Zola, a proper SSG, was a chance to fix many of it's previous technical shortcomings. These improvements compound, even when the effect is not a drastic unrecognizable change.

Scraping

2025-09-23T00:00:00+00:00

Web scraping is the process of automatically accessing and collecting information from websites using bots. It's essential for web search, LLM recommendations, live web monitoring and so on, but it can have a downside of generating unwanted traffic and costs. For this reason some websites use various anti-bot measures and in the web content arms race the scrapers unevitably grow more complex.

There are three main ways to recognize and mitigate bot traffic:

displaying captchas for risky browser fingerprints
blocking or rate-limiting abusive IPs
making scraping cost more than it's worth

For the dynamic web it's not feasible anymore to simply fetch the website text and parse it. Often there is some JS involved that populates the website content. Some of the aforementioned anti-bot measures can be involved too, thus the entire browser has to be utilized in the scraping process to be effective, circumvent protections, and render the target content properly.

There are two main browser automation toolkits: (1) selenium and (2) playwright. Both provide some facilities to make the browser appear more like a real user. For selenium there is undetected-chromedriver and for playwright there are extensions such as puppeteer-extra-plugin-stealth. Choice between the two pretty much comes down to personal preference and external tooling support.

To avoid IP blocking scrapers have to rotate their addresses regularly, utilizing various proxy services. Some of them are pretty shady, for example residential and mobile IP proxies, like the ones from brightdata (previously luminati). The Evil™ Corp pays mobile devs to include their Evil™ SDK within their app that allows other people to use YOUR device as a proxy. Of course it all has to be obfuscated somewhere in the terms of service, but nobody reads a fucking 200 page TOS for a stupid mobile game. The malicious Evil™ SDK ends up in many places where you don't suspect it: mobile apps, browser extensions, games, the .exe you downloaded last time, and many more. The more you know...

Even though it's not hard to build out a scraping setup like this by yourself, it would be much more convenient to have it all bundled in one service. Unsurprisingly, there's a lot to choose from: scrap-this, scrap-that, com, io, dev, whatever. There are even some designed for plug-and-play use with LLM tools, for example: browserbase, firecrawl.

Last time I used browserbase to scrape a list of about 200 ecom websites and extract contact info from them. Data like this can later be used for marketing campaigns, so called "lead generation", robocalls, email spam, phishing, or other cyberattacks. It's easy with browserbase, because I don't have to grep for contact links and emails, I can use stagehand and have LLMs locate this data for me, navigate the page even, and in the end extract only what I need. Browserbase exposes playwright API for their browser session, so I can also take screenshots and shapshots of these websites to analyze them more throughly later. Theo's recommendation once again comes in clutch. I hear more and more about LLM-based scrapers in the browser and other tools, so I suspect this is only going to get easier with time.

I think the third way of mitigating bot traffic, making it financially unfeasible, is currently the best solution. I am very much rooting for Anubis to win in the battle against the bots and even out the cost of serving the content with the cost of scraping it. There must be a better and less hostile way to obtain a good data set than just wild data harvesting. And there must be better ways to protect your services from it.

LLM agents

2025-09-21T00:00:00+00:00

Getting a little deeper into the good, the bad, and the things I miss after playing around with LLM tools in the terminal for some time now. In my previous post I gave an overview of some tools built around LLMs specifically for programming and which ones I ended up using the most myself. This time I briefly explain how such things are built and then lay down some ideas on what could be improved to make LLM-powered tools aid my processes even better.

By a "tool" we could mean many different things, like:

LLM agents (tools around LLMs)
programs that the LLM agent calls (tools for LLMs)
other programs that we use
plugins for other programs

Let's walk through the anatomy of an LLM agent first. It will help us see where each tool type fits in the whole thing. In general agents are built around an inifinite loop with three steps: input (prompt), output (inference), actions (tool calls). We will refer to the code that manages this loop as the LLM harness. It works kinda like a programming language interpreter. First the model is informed about capabilities of the harness - for example it can get a plain text explanation of provided tools along with expected input/output formats for a proper call. The model might be fine-tuned for a more complex format. Sometimes it can be overfit for a particular set of tools, or harness capabilities, which causes interoperability issues. When harness picks up a tool call pattern in the LLM output it makes the call with data passed by the model and in the next loop interation feeds that to the LLM as input. When there are no more tool calls to process, the harness goes back to asking the user for another prompt.

Agent takes actions by executing tool calls, which can be implemented as:

function calls - using code straight in the agent harness, similar to how naive interpreters work
tool calls - this is a general term, but it can also mean invoking external commands on the local system - such as bash, grep, find, head, linters and so on
MCP client calls - using a protocol or framework for making a call to an external server, which shifts responsibility of implementing the former two to the MCP server developer

Model Context Protocol (MCP) functions as a middle man between the place where tool calls are invoked (client, the LLM agent) and where these calls are executed (the MCP server, integrated with some resource). Personally I don't find a need for it - local agents already have shell access with all the tools they could ask for. I'd rather enhance the capabilities of my shell with additional CLI tools. Sure there are some things that could benefit from restructuring I/O for LLM consumption and that's where I'd consider plugging MCP if I wanted to connect LLM to the thing. I think it's a good idea as a conversational interface for your services. I would also consider implementing MCP as a client if I was making my own LLM agent from scratch.

To finish off my quick agent anatomy course we have to discuss context management. LLMs have a limited context window they can process - information that doesn't fit in the context is not considered during inference. For the most popular models that window is still rather small if you consider how big source files can get in long running projects. This is the hard part. If you don't manage the context well the agent will quickly forget what it was doing. Good prompting beats fine-tuning for general tasks - and this is also part of context management. First the LLM harness needs to load up tool definitions; then specific user-defined rules from AGENTS.md. Often the LLM agent tool also comes with baked-in system prompts or prompt templates that tell the model how to behave for the most effective outcome. You will easily find entire prompt template collections online (for example danielmiessler/fabric), and AGENT.md or cursorrules repositories. "Teaching the LLM" is all about effective context management strategies.

Here's one example - it's very effective to include relevant resources in the LLM input - this strategy is called retrieval-augmented generation (RAG):

the user can specify resources or include them directly in their prompt
the harness can go fetch relevant resources based on user's prompt and include them as additional input
the model can request related resources through a dedicated tool call

RAG is one of the things I like the most about this LLM rush. Embeddings and vector databases can be used for sweet semantic search with applications beyond silly chat bots. RAG also allows the model to cite sources with higher reliability. I like how the need for compressing technical documentation for LLM consumption resulted in some good dense text documentation which is very accessible and helpful for human learning. We got new scrapers, parsers, better OCR, search, and so on - mainly to make documents easier to digest for LLMs, but accidentally also making many resources easier to process by hackers like myself.

My point is that some tools for LLMs are really cool for people too. Many of them would fit very well in a regular development flow. Nothing prevents you from calling LLMs as a CLI tool or giving your LLM agent the ability to spawn sub-agents. C0gniti0n the Devin company suggests it might not be the best idea, but who the hell would listen to those guys anyway, right? On the other hand Anthr0pic uses subagents in Claude Code and describes LLM workflows that involve multiple workers on their blog, which seems to be a pretty good resource: https://www.anthropic.com/engineering/building-effective-agents

You might be eager to write your own LLM agent now. You will unevitably find many libraries that claim to be the best for this task, but let me just remind you - aichat proves you don't need any external libraries for handling inference from an API. If you are to pick one API to integrate - make it OpenAI-compatible API and use LiteLLM as a proxy for your target model. You will get logs, stats and cost tracking for free. You can even plug GitHub Copilot as a provider. Of course if you prefer to use LangChain or whatever else - it's up to you. In the end you might still find yourself using more purposeful tools, such as aichat or Simon Willison's "llm" LLM tool collection as building blocks or to enhance your own flow instead of replacing it with agents.

People have varied opinions when it comes to LLM agents, how they should be run, managed, etc. This blog post is a good overview and will probably give you some ideas: https://shmck.substack.com/p/claude-code-framework-wars

Finally we get back to where we started this post. I think the missing piece for LLM agents right now is easy knowledge retrieval and self-hosted grounding. This is mainly because managing context is difficult and mostly left for the user. If I were to address this issue, I'd use a specialized agent to help select the scope for the coding agent and prepare a well-formated prompt. I would also use RAG to include additional context about target of the change. Instead of making the agent grep for function usages and references, it could be included in the context. Closer integration with treesitter would also be an interesting change. The LLM wouldn't have to target files and walk the filesystem anymore - it could target specific classes and functions directly, and have more information retrieved from the vector database after one initial sweep of the codebase. You could also chat with the codebase through RAG after that one initial sweep. This might not be the best approach, but I'm curious to see how it would work in practice.

Anyway, to recap - an LLM agent is basically a dumb LLM calling a bunch of dumb tools in a loop. Surely that's worth millions of dollars if not more?

madgit

2025-08-30T00:00:00+00:00

I decided to give a lightning talk during Cebula Camp 2025. The title I submitted was: mad git - custom commands for fun and profit. These are my notes describing more or less what this is about. For people coming here right after the talk - hi! Sorry the talk was scuffed, public speaking is crazy stressful. I'm glad you've decided to check out my little internet corner. I hope you like it here.

Let me first introduce myself in a single sentence and then I'm gonna move on to the problem statement. This is not going to surprise you: I'm a programmer, dealing mostly with backend webdev and cybersecurity.

Sometimes I want to read (IYKYK) some code. A lot of code. I go to GitHub and see a user or organization and rarely ever do I end up interested in only one repository. I know we all love git, I'm sure we all love GitHub even more. Maybe instead of clicking through pages of repos I could just clone them all and browse the code directly? But.. what if I want to grab some repos from GitLab (or Forgejo) too? Oh, and I also want to keep my copies up to date!

At this point most people would just G00gle the answer (and figure out there is an official gh cli tool, wtf?) or generate a bash one-liner with GeePiTy and move on with their lives. But you know, I have that weird muscle memory, so I always end up opening the network tab in dev tools first. Little 'curl | jq | git' in a for loop and we're good.

This is exactly what I did and for a while I was happy. In a few lines of bash I had a script to clone a list of git links. Updating was super easy too, by the way. I was actually using it more than expected. I'm sure many of you had to write a similar script: walk directories, git pull in all of them. Very easy, very useful, feels good.

This is not all I wanted to do with these repositories. It is however the first basic building block that I would call over and over again... If I'm gonna call it more than once, then I might as well... (Maybe you already see where this is going.) Might as well rewrite it in Rust. It's nothing fancy... But I still put it on crates.io and AUR for convenience. I called the tool madgit - it solves exactly the two of my problems that I stated in the intro. It also allows me to call any arbitrary command in each git repo from a selected directory. There's many directons this could further go... No promises tho! Check out the project README if you're curious.

That's cool - I solved my little problem and published a tool to address it. We all have little scripts and personal tools like that, they are often underrated productivity boosters. Sometimes they seem so insignificant that we don't bother sharing. It seems like it's just not worth the time to clean it up, document and push out. If you're one of those people - publish with no warranty, but just do it.

This got me started on a journey to discover more custom commands made by other people. I know there are also other git wrappers, UIs, TUIs and whatever. I don't want to make a complete list, I just want to share some cool stuff I found. There are two ways that I know of to modify git subcommands without writing a wrapper: (1) alias in the config (here is a gist tutorial for this) and (2) having a git-[subcommand] executable in your $PATH. I compiled a list of some custom commands below, all of them are available in AUR. There is a pretty good description of how this works here: https://nathancraddock.com/blog/custom-git-commands/

There is a lot of tools for managing branches and encrypting files, I skipped those. I also skipped demos and examples, sorry for that, but at least the resources are there.

STATS

git-cal - github like contributions calendar on terminal
git-heatgrid - visualize git commits as a calendar heatmap
git-hours - count the time spent on code via git
git-punchcard - show punchcard graphs with git commit activity.
git-quick-stats - simple and efficient way to access various statistics in git repository
git-standup - recall what you did on the last working day
git-summary - script to print a summary of git author contributions

CHANGES

git-chglog - git changelog generator
git-delta - syntax-highlighting pager for git and diff output
git-file-history - easily view the history of a file in a git repository
git-playback - a git command to play back or step through the history of a git-controlled file.
git-when-merged - determine when a particular commit was merged into a git branch
git-who - git blame for file trees

REBASE

git-fiddle - edit commit messages, authors, and timestamps during git-rebase
git-filter-repo - quickly rewrite git repository history (filter-branch replacement)
git-imerge - incremental merge for git
git-machete - git repository organizer & rebase/merge workflow automation tool
git-revise - a git subcommand to efficiently update, split, and rearrange commits

IDENTITY

git-identity - manage your identity in git
git-switcher - git profile switcher switch between your git profiles easily
git-user - git plugin that allows you to save multiple user profiles and set them as project defaults

CLEANUP

git-debubble - clean up merge bubbles in a git repository
git-delete-merged-branches - command-line tool to delete merged git branches.
git-fresh - keep your repo fresh with one command.
git-gone - prune stale local git branches
git-superprune - a git command to prune remote branches that have already been merged
git-trim - trims your git remote tracking branches
git-warp-time - reset file timestamps to repo state

TOOLBOXES

git-commands - fuzzy search git commands with fzf
git-extensions - simple handy extensions for git
git-extras - repo summary, commit counting, repl, changelog population and more
git-toolbelt - a suite of useful git commands that aid with scripting or every day command line usage
git-town - generic, high-level git workflow support
git-x - cli extensions for git that simplify common workflows
git-xargs - making updates across multiple github repositories with a single command

Checking out those git custom commands feels like figuring out nvim plugins again. Maybe some of this is helpful to you. At the end of the day... It's not about the repos we cloned, it's about the journey... It's about the "shitty scripts we threw together and then immediately thrashed" along the way.

Happy hacking everyone!

Deployments

2025-07-27T00:00:00+00:00

Imagine a user wants to use your app, so they look it up on the internet and find your domain. They go to your domain and you want to serve them your webapp. For any of that to work, you need a few things prepared first on the deployment side: (1) registered domain, (2) DNS records set, (3) have a network-accessible server or a proxy/load-balancer/firewall, (4) host or container for you app. Only if all that works smoothly can you resolve the request with your app and reply with a response. In this post we're gonna look at some solutions for steps 3 and 4.

Your app has to run somewhere, it has to be executed on some machine that is also in some way accessible to your users. The process of putting it there, especially for production use, is exactly what we call deployment. There are a few approaches to getting that done.

The traditional approach is to spin up a server, either on real hardware by yourself or as a rented VPS, install all the software you need on that server, and then expose it to the internet. This is a very good and easy approach. You get to understand what happens at every step of the way. The pricing is clear and limitations of this setup are rather obvious. It's difficult to scale, Continuous Deployment can be tricky, and sharing the same server for multiple projects is not necessarily clean.

To build on top of that, you could use containers, for example with docker compose, to run multiple projects on one host. The advantage of this is that your system configuration is now confined to a container, so there can be no issues with shared libraries or conflicting dependencies between different projects. This is a sweet spot for ease of use with small projects. The setup is usually smooth and once it's done - deployments become very easy. All you have to do is rebuild and push your container, then bump the image tag on your host to re-deploy. Scaling would be limited to the power of your host, unless you apply another solution...

You could use docker swarm mode, which lets you utilize multiple hosts clustered together to serve docker images cooperatively. This is all fine and you will find many different solutions built on top of that. Some neat self-hosted solutions for deployment management with Docker Swarm I would like to mention: Co-op Cloud, CapRover, Coolify, Dokploy, Swarmpit. The original idea for this post was to go through these, compare them, and perhaps conclude on what I want to use more. Unfortunately, they are all too cool, so instead I suggest you check them all out by yourself. Personally I use Coolify and Co-op Cloud the most.

There is another common approach to deployments: using Kubernetes, or k8s for short. It's probably the best solution for modern distributed computing, scaling at a different magnitude. It was designed and released by Google and is now widely supported in the world of enterprise open-source; notably with RedHat's OpenShift products being built on top of it. Kubernetes is a beast that deserves a dedicated blog post or even a few. This solution is complicated enough to create a dedicated role in the industry. DevOps, which was originally meant to be Development+Operations, became more of a modern infrastructure administration role centered around a software development model spawned from the adoption of Kubernetes for deployments. Cloud, microservices, IaaC, CI/CD pipelines, containers, and more. These topics were heavily hyped up not too long ago - it was a big one just before LLMs and cryptocurrencies took over the news. All of it is easy enough for a motivated developer to understand and operate, but time-consuming and annoying enough to warrant getting another person employed just to take care of it. Many places build out their own pipelines and custom UIs just to make these things easier to operate.

For certain workloads there is a better way. With each approach discussed we've been abstracting away from the bare metal machine running our code. The next step is to go serverless, or in other words, to completely forget about the underlying servers running our deployments. Kubernetes, Docker Swarm, VPS administration - with this approach none of these are our concern anymore. Unlike previous options, this one is pretty much always bound to some managed solution. Our app code will run on ephemeral virtual machines spinned up on demand by our cloud provider. That way the infrastructure we rent out can scale anywhere, even down to zero (free beer!). Some interestng providers, besides the big three of cloud, that can offer this are: Cloudflare, Fly.io, Netlify, Vercel. You will likely need a managed database to match. Lately I really like to use Turso for this, so even though it can be kinda a controversial choice, I'd still recommend that. I'm skipping over the details of "serverless functions" and instead I leave that part of research as an exercise for the reader. Serverless is really great if you don't want to get deep into the infrastructure rabbit hole and still want to deliver good and scalable experience for your users. This is what truly makes developer's life easier. We get great UIs, great management options, logs, easy and fast rollback features, easy setup, scaling, all the good stuff. If you need a better explanation of what serverless is and how it's different from a k8s deployment, here's a really good video by Theo (t3.gg) that I highly recommend watching: https://www.youtube.com/watch?v=_VQl_HTk9PM

To summarize, the common options are:

deploying software directly on a server
deploying through docker on a server
using docker swarm, perhaps through some other UI
using kubernetes, again perhaps through a dedicated UI
using serverless deployment solutions

And finally here's my recommendations on what to pick:

If you are working on a modern webapp - go with serverless deployment using Cloudflare or Vercel.
If you're working on a research project, especially in early stages, consider the simplest of options: 1 and 2.
For almost anything else Docker Swarm directly on your VPS is probably the right choice, especially with additional open-source tooling. Personally, that would be my preferred choice.
If you need to scale over multiple machines or run a big enterprise - Kubernetes is for you.

Happy DevOpsing!

LLM tools

2025-07-05T00:00:00+00:00

Back in January I finally gave Cursor IDE an honest go and it sold me on the idea of finally using LLMs for programming. Since then I've tried multiple different tools, and even though I still hardly use generated code in my projects, I now see many ways in which this technology can be useful day-to-day.

The most basic tool with the widest adoption is a chat interface. Think: ChatGPT, t3.chat, aistudio. You get to talk to your model of choice directly, ask questions, give out tasks, and so on. I find it especially useful for summarization and search. Summarization is one task that LLMs are actually good at. Proper web search options, such as deep research and search grounding, which provides links to other sources, go a long way too. Searching for something using vague context description is great for exploring. Unlike a traditional search engine, you don't need a very specific query anymore.

Generating self-contained code examples is where I find most value of LLM tools for software development. Small code snippets are easy to verify and save a lot of research time. Instead of reading through docs and library source code, you can generate a personalized example and see if it works for your problem. I usually choose to adapt and transform the code myself, so I still get to do some thinking. The LLM is just there to help me find a working solution faster. Some people choose to go further...

Around February I went through a list of VSCode extensions and VSCode-based editors which enable comfortable LLM usage for development. All these options were really very similar. Open-source: AIDE, Cline, Continue, PearAI, Sourcegraph Cody, Tabby, Void. Proprietary: Codi, Cursor, GitHub Copilot, Qodo, Supermaven, Tabnine, Trae, Windsurf. I was looking for something I could use on my own terms: offline and with open-source models. Even with paid tools, I was quite disappointed. With Cursor everything goes through their servers before hitting any model, so they clearly don't care about self-hosted models. From the open-source selection - Continue seemed interesting. It even got featured on Ollama blog. After trying it out myself it quickly felt lacking in comparison to a full-blown editor, but so did other solutions I tried. Perhaps trying to fit an LLM in the constraints of an existing editor was not such a good idea afterall.

Cursor does many things right and accomodates a wide range of users. It's very good at picking up context and displaying suggested changes. You could think that is exactly what makes it distinctly better than the rest. I certainly thought so at first. There is more: Cursor has amazing autocomplete, well integrated inline editing (Ctrl+K), and a Composer mode for conversational development flow closer to a chat experience. That's like fitting three tools into one, each covering a different prediction distance, each of them scratching a slightly different itch. I believe that is what makes the Cursor experience so good. Out of these three, autocomplete was the first application of LLMs for coding (the other two require instruct model tuning, which came later). It covers the shortest prediction distance, so it's usually rather accurate. It helps with tedious and repeatable edits and typing out boring parts of the code. Then again, so does learning vim. I take pleasure in typing and do it fast enough not to really need completion suggestions for the parts that I already know how to code. When it comes to parts that I'm not yet sure about, I would rather stop and think the problem through, than have an invasive completion suggestion pop up to skew my direction of thinking. Anyway, out of the three editing modes, only this one really needs to exist within the editor.

Inline editing is a pretty cool idea. It can both handle your tedious edits when you don't want to bother coming up with a good macro, and can handle small-scale changes, generating snippets of code in-place. It is also not very invasive - it only works when the user requests it. Finally Composer, with the conversational approach, has the furthest prediction distance. The LLM has to find a proper place to make the edit and then attempt to make a prediction far into the future, to generate a change that is possibly very distant from the present code. It makes this mode prone to mistakes. Generally speaking, the more autonomy you give to the LLM and more complex the instruction, the lower quality and accuracy goes. These two modes don't have to live in the editor. Let's discuss another family of LLM tools - agents.

Meet Aider - the first proper LLM agent for the terminal. It is Apache licensed and written in Python. Aider introduces a few chat modes: one lets you quickly ask questions about your codebase and plan out changes, another lets you apply changes to files right in your filesystem, and then there is a third one that kinda combines the previous two. It's kind of like Cursor's Composer, but better. It works with many different providers and any LLM model you could think of. It does conventional commits after each change, so you can thoroughly and comfortably review them. Aider also has a watch mode, which reacts to file changes in a repository and acts on special comments that you can place anywhere. For example you could write a function declaration and right above it a comment "Implement this function. AI!" and Aider should create a matching function definition or fill out the function body. I like the Aider workflow even more than the Cursor experience. The ability to use different models and to seamlessly switch between them, different tools, chat modes - all of this makes Aider stand out as one of the top LLM tools for development. There is a model benchmark section in Aider docs, in case you're wondering which LLMs give best results: https://aider.chat/docs/leaderboards/

A complementary terminal tool, also very powerful and flexible, is aichat - written in Rust and available in official Arch Linux extra repos. I haven't used it much yet, but I see huge potential for creating quick terminal commands and personal workflows on top of it. I like two concepts in particular: user-defined roles, a collection of system prompts stored in a directory; and macros, which expand to longer bits of pre-defined text which you can re-use in your prompts. Configuration options are very extensive. There are also some web interfaces that aichat can expose for you, like an openai-compatible api proxy, very useful for testing and development.

After Aider, all major players released their versions of LLM tools for the terminal. First we got Claude Code, some people swear by it, but it is closed source and closely tied to Anthropic, so no way in hell I will ever run it. Then OpenAI Codex came out, open-source project in TS with an ongoing official Rust rewrite, unfortunately completely unusable in my experience. Finally, recently released Gemini-CLI, also open-source and in TS, has a good vibe, but lacks basic features compared to alternatives and is also tightly coupled with G00gle's LLMs. To be fair, none of them are worth wasting time on. If you're looking for something that's more fresh and simple than Aider, I have another tool to recommend.

Lately my go-to LLM terminal tool is sst/opencode. This FOSS tool has the cleanest interface I've seen so far. You can't deny it beauty and simplicity. It's also written in TS, but very fast and responsive. I wish it had more advanced features exposed to the user, but defaults are more than good enough, and you can still plug in any model with almost any provider. I often prefer it over opening the web chat interface (I still do that for search). There is a handful of tools built-in and included for the LLM to use: running general shell commands, fetching web content, keeping track of a TODO list (works great for making the LLM implement everything you ask for step-by-step), searching for files by pattern or with grep, basic file operations like listing, reading, writing, editing. That's really all you need for effective agentic work and opencode is great at it.

Having so many LLM agent tools, you may think about running multiple agents in parallel for easy agent/model comparisons and quick re-rolls of changes per prompt. That's exactly where uzi comes in. It's a fairly simple project that lets you orchestrate local agents. It can run multiple agents in separate tmux sessions, each working in their own git workspace, on a separate branch. You can control multiple at once by passing stdin directly through tmux, all managed by uzi. Personally I'd take a different approach and use Docker or firecracker VMs for each agent, but then again - that might be more hassle than it's worth.

Regardless of the interface, there's still a serious issue with LLM generated code. The way LLMs write makes single files read really good, but only in separation. Whenever changes are contained within one file, one class, one function - the results are usually more than satisfactory. Once the logic is spread throughout different files, that's when everything falls apart. Again, in separation all new changes look fine, but they lack any design. The LLM does not really think and that lack of thought shows greatly in resulting interfaces, object relations, module separation, and all those other design decisions that human programmers with experience make intuitively on a whim. Generated code looks good on surface, but is very hard to work with later on. LLM agents are also not fully capable of utilizing existing code well for the same reason - they don't have complete enough understanding. Best they can do is explain bits of code, explain their goal out loud, and if those two match, make a decision to use a certain method. LLMs are thus happy to repeat themselves - it's easier for them to process and is therefore cheaper. It's far from how real programmers work. Real programmers don't use Pascal. Utilizing LLM tools well is a new skill to learn. They can enhance our work, but they need a competent human to drive them properly to really be useful.

There is a lot more to talk about: building tools around LLMs, building tools for LLMs, dense documentation, retrieval-augmented generation, fine-tuning, big models calling specialized models, transpilation, interesting tasks for LLMs... I will most likely come back to these topics later. For now - I'm excited to still have a lot to learn.

AccelProfile

2025-02-17T00:00:00+00:00

Default mouse acceleration settings on X11 are anoying as hell, especially when you just launch your i3-wm and realize mouse movements are way off. On most DEs you can change it from within settings, but with a WM you're on your own.

ArchWiki article on this topic: https://wiki.archlinux.org/title/Mouse_acceleration

Quick fix: /etc/X11/xorg.conf.d/40-libinput.conf

Section "InputClass"
	Identifier "libinput pointer catchall"
	MatchIsPointer "on"
	MatchDevicePath "/dev/input/event*"
	Driver "libinput"
	Option "AccelProfile" "flat"
EndSection

The second fix is for touchpad tap to click. Whether I like to have it enabled depends on a device.

Section "InputClass"
	Identifier "libinput touchpad catchall"
	MatchIsTouchpad "on"
	MatchDevicePath "/dev/input/event*"
	Driver "libinput"
	Option "Tapping" "on"
EndSection

You're welcome.

Window$ setup

2025-02-04T00:00:00+00:00

How to make this piece of shit usable for a UNIX hacker, or even usable at all? The default answer used to be cygwin, but more recently it's been WSL.

My setup for "work" on "Window$" would look more or less like this:

WSL2 (for the Linux kernel, otherwise WSL1 runs through a compatibility layer)
Debian
bash
tmux
neovim
alacritty

You can set up alacritty to go right into WSL2+Debian+bash on launch by default. Consult alacritty docs for details.

WSL exposes Window$ filesystem, so there's really no need to leave the terminal ever again, but if you have to - graphical programs also work under WSL. Just launch alacritty full-screen and enjoy the familiar and comfy Debian. In terms of usability it's a HUGE improvement, but there is one disadvantage: RAM by default is split 50-50 between WSL2 and host, which means Window$ will still eat up half of it, unfortunately.

If you somehow can't get admin access on the machine to install whatever you want, there's a different setup you can try.

From M$-Store you can install:

Window$ Terminal
Powershell (7)
neovim
python

Window$ Terminal is pretty good and Powershell 7 is better than cmd.exe or Window$ Powershell 5. Not ideal, but you can get by if you stick to neovim-based workflow. You can try to build out a different kind of shell with Python-based xonsh, but results may vary.

There are some tweaks that can also make the WM experience a little less painful.

Settings to change:

Disable personalized ads, disable news.
Change Alt+Tab behaviour.
Change window snapping behaviour.
Move Start Menu to the left.
Disable Search Bar.
Disable Copilot.
Disable notification pop-ups.
Hide tray icons.
Create more virtual desktops.
Disable default lid action.
Disable desktop icons.
Pin terminal to taskbar.
Display taskbar on main display only.

Using this OS is truly an exercise in patience. My x220 (over a decade old laptop, CPU only has 2 cores) with Arch Linux felt much snappier than a modern top-end ThinkPad running W11. On paper the host machine was multiple times more powerful, but in practice it felt multiple times less performant, with huge loading times and latency for almost any basic action. Only in WSL2 terminal felt fine, but I'm pretty sure installing any distro straight on the device would feel even better. With a native GNU/Linux install you could also leverage a proper window manager... User experience on Window$ is absolutely horrible, but still there's one thing I actually liked about this system and that is virtualization. WSL2 integration is done fairly well and GUI integration especially is quite impressive. Guest OS runs great, no complaints there. Hyper-V overall is pretty cool, although I find no reason to pick that over Linux KVM. It's nice to see at least some things done well over on Window$.

With each release since Window$ 7 (hell, maybe even XP) the experience for a consumer has only been getting worse. Window$ Server holds up better, because it doesn't include as much bloatware by default, but even then - it's not great. If you're stuck with it then I truly pity you, but if you just can't decide what to switch to - I wrote a post about Linux Distros a while back, maybe it will help you decide where to start.

Good luck!

Python tooling

2024-11-11T00:00:00+00:00

Python tooling currently is a mess. Other modern programming languages, like Go or Rust, have a unified standard tooling to cover all bases. Meanwhile Python suffers from tool overload. For every aspect of working with a project there's at least a few different tools you can choose from. While development tools are greatly improving, it seems like project management has gotten a little more complicated.

Good overview is presented here: https://alpopkes.com/posts/python/packaging_tools/

Another interesting insight is presented here: https://chriswarrick.com/blog/2024/01/15/python-packaging-one-year-later/

Here's a "very short" list of development tools for Python:

pip - managing python packages
venv - isolated virtual environments
pyenv - managing python versions
setuptools - building packages
twine - publishing packages
poetry - complete project management
hatch - project management, again
rye - different project management, again
maturin - project management for Rust packages
pyinstaller - freezing into binaries
pytest - testing framework
black - simple linter
isort - import linter
flake8 - extensible linter
ruff - linter and code formatter (+LSP, very cool!)
autopep8, pyflakes, pylint, ... - more linters
mypy - type checker
jedi - autocompletion, static analysis
pyright - static type checker (+LSP)

For some tools there might be more alternatives available too... In short - tool overload.

Let's first cover dependency management, then project management, and lastly development tools like linters and static checkers. I'll mention my personal choices for each.

The most basic tools are pip (a package manager) and venv (virtual environment to contain and separate packages). They are very useful and probably enough for simple development purposes, but not quite enough for building and distributing packages. You can use them along with 'requirements.txt' file for project dependency management.

Now for some people, getting the Python version right is also an issue. I don't see why you wouldn't just use the lastest version offered by your distro, but I guess some poeople are still stuck with inferior operating systems or are constrained by the deployment environment. This is where pyenv or conda comes in. I haven't used either much myself, but I mention them here for completeness.

There used to be a common and simple tool for package building - setuptools - but right now it's quite a mess. You would set your metadata and configure the build in a 'setup.py' file, then you could build the project by executing that file. This offered a lot of flexibility, since a regular python file realistically can contain any code you would need. The downside is that it's difficult to parse for external tools.

Then 'pyproject.toml' configuration got introduced with PEP-518.

Many build tools make use of the 'pyproject.toml', including setuptools, which is not as dominant anymore. Recently my favorite is poetry. It's a complete project management tool, which by now plays very well with 'pyproject.toml', and takes care of pretty much everything: dependency management, virtual environments, package builds, package distribution, etc. I find it very easy and pleasant to use. It doesn't fuck up my project structure and nicely integrates with other tools, for example pytest.

Another project management tool, rye, embraces a slightly different vision. It also covers most bases, but it uses a specific set of specialised utilities to accomplish that. Their choice of tooling is opinionated, but very solid and also quite interesting. For example: they use uv instead of pip. I believe rye, uv, ruff were all written in Rust by the same original author (they are all part of astral.sh). On their "philosophy" page you can read how exactly rye is different from other project management tools and why.

PyInstaller is useful when you want to distribute your program for a platform that might not have the appropriate python version already available in the PATH. You freeze your project along with dependencies into a single binary file and run it on your target. It's pretty cool for some use cases.

When it comes to development, Python doesn't have a strict style. However, especially when you're working with others, you would probably like to automatically enforce a uniform and clean style that doesn't make your eyes bleed. That's exactly what a linter does. There's more than a couple of those you can choose from for Python. Black is a pretty simple yet solid choice. Just for the purpose of sorting your imports correctly there is isort. Extensible and powerful flake8 is a popular linter too. It's very good, but not very fast. That's where ruff comes in, offering parity with all the three at faster execution time. Speaking of Python style, the classic style guide is PEP-8.

Python is a dynamic language, which means that variables don't have strictly assigned types. Nonetheless, we can enforce type consistency through static analysis with a type checker such as mypy. Static typing makes it easier to catch some errors before reaching the faulty program state. Static analysis can extend beyond that to catch all sorts of mistakes and provide helpful hints about the correctess of the program. Good tool examples are pyright and jedi, which can also provide completions and help effectively browse the code through LSP integrations. You don't even have to provide any type hints yourself to make good use of a static analysis tool such as pyright. In case you want to get a refresher on Python type hints anyway, they were described in PEP-484.

LSP is a protocol that allows for integrating tools with IDEs or editors such as Neovim. LSP tools help out a whole bunch during source code editing. I myself use aforementioned ruff and pyright that way.

In the end you have to reach for multiple tools to get what cargo gives you by default for Rust. For now poetry+ruff+pyright feels comfy enough to me, but it was quite a journey to get there. I'm sure Python tooling is going to evolve further, hopefully for the better.

OpenVPN

2024-11-10T00:00:00+00:00

As I mentioned in my previous post on VPNs, it's pretty easy to set up an OpenVPN server. This is my quick setup guide and pretty much what I use on my Debian VPS. I have PKI set up on a different local machine and that's where I generate my client configs. Once set up you only need a single .ovpn file for a client to connect.

Config files for openvpn server go to: /etc/openvpn/server

Files you want to have there:

avpn.conf
ca.crt
avpn.crt
avpn.key
ta.key

The 'avpn' part is simply a name for my config, so you can change it with no problem, just be consistent.

Install the proper package, move config files, then enable 'openvpn-server@avpn' service. Update your DNS entries for the VPN access too. Setup with cloud-init and my server conf follows.

cloud-init

#cloud-config
package_update: true
package_upgrade: true

packages:
  - openvpn
  - ufw
  - ca-certificates

runcmd:
  - ufw allow 1194/udp
  - ufw allow 22/tcp
  - ufw enable

  - |
    cat <<'EOF' > /root/openvpn-ready.sh
    #!/bin/bash

    CONFIG_FILE="/etc/openvpn/server/avpn.conf"

    if [ -f "$CONFIG_FILE" ]; then
      systemctl enable --now openvpn-server@avpn
      systemctl status openvpn-server@avpn
      echo "OpenVPN service enabled"
    else
      echo "Please copy your configuration to $CONFIG_FILE and run this script again."
    fi
    EOF

  - chmod +x /root/openvpn-ready.sh

final_message: "Cloud-Init finished. Copy your OpenVPN configuration files to /etc/openvpn/server/avpn.conf and run /root/openvpn-ready.sh"

avpn.conf

port 1194
proto udp4
dev tun

ca	/etc/openvpn/server/ca.crt
cert	/etc/openvpn/server/avpn.crt
key	/etc/openvpn/server/avpn.key

topology subnet

server 10.8.0.0 255.255.255.0
keepalive 10 120

dh none
ecdh-curve secp384r1
tls-crypt /etc/openvpn/server/ta.key
auth-nocache

cipher AES-256-GCM
data-ciphers AES-256-GCM
auth SHA512

persist-key
persist-tun

verb 5

client-to-client
explicit-exit-notify 1

Below is a script that I use to generate my client keys and .ovpn config files. Omit spaces within tags.

add-vpn-client.sh

#!/bin/bash

name="client-$1"
echo $name

export EASYRSA=$PWD
easyrsa gen-req $name nopass
easyrsa sign-req client $name
mkdir $name
cp $EASYRSA/pki/private/$name.key ./$name/
cp $EASYRSA/pki/issued/$name.crt ./$name/
chmod go+r -R $name

echo "writing .ovpn config"
touch ./$name/avpn.ovpn
cat >> ./$name/avpn.ovpn << EOF
client
remote vpn.wykwit.pl 1194
proto udp4
dev tun

ca   [inline]
cert [inline]
key  [inline]
tls-crypt [inline]

ecdh-curve secp384r1
cipher AES-256-GCM
auth SHA512

< ca >
EOF
cat ca.crt >> ./$name/avpn.ovpn
cat >> ./$name/avpn.ovpn << EOF
< /ca >

< cert >
EOF
cat ./$name/$name.crt >> ./$name/avpn.ovpn
cat >> ./$name/avpn.ovpn << EOF
< /cert >

< key >
EOF
cat ./$name/$name.key >> ./$name/avpn.ovpn
cat >> ./$name/avpn.ovpn << EOF
< /key >

< tls-crypt >
EOF
cat ta.key >> ./$name/avpn.ovpn
cat >> ./$name/avpn.ovpn << EOF
< /tls-crypt >
EOF

Setting up PKI with Easy-RSA is described very well on ArchWiki: https://wiki.archlinux.org/title/Easy-RSA

Scientific method

2024-11-09T00:00:00+00:00

Scientific method is OP and can be applied to many more things than just pure science. Methods of engineering design are very similar, only applied to a different problem. It's really a flexible and useful framework to inteligently approach any problem and work out a satisfying conclusion. For example TDD can be seen as the embodiment of scientific method applied to software engineering, where instead of modifying the hypothesis to match your observations about the world, you modify the world to make observations match your hypothesis. I find it useful to remind myself steps of the scientific method from time to time.

| step | scientific method | engineering design |
|------|-------------------|--------------------|
| 1    | question          | problem definition |
| 2    | observation       | research           |
| 3    | hypothesis        | planning & design  |
| 4    | experiment        | prototype          |
| 5    | analysis          | evaluation         |
| 6    | conclusion        | improvement        |
| 7    | report & repeat   | report & repeat    |

Distros

2024-10-07T00:00:00+00:00

Linux is a kernel. Linux-based operating systems bundle all kind of software, mostly open-source or FOSS (often coming from the GNU project), to create a friendly environment for the user. They are commonly called Linux distributions or "distros" for short. The big difference between them is in how exactly they distribute the software. We use packages and package managers for that.

Debian family uses .deb packages and the "apt" package manager.

RedHat family uses .rpm packages and "dnf" or "yum" as a package manager.

Arch Linux uses .tar.zst packages (compressed tarballs) and the "pacman" package manager.

There is a list of common operations along with a comparison of commands for major distros in the pacman/Rosetta article on ArchWiki: https://wiki.archlinux.org/title/Pacman/Rosetta

There is also a small difference in what distros bundle in their installations by default. Most of the time that's not very important, since you can customize any distro to operate, look and feel exactly the way you like. From user perspective most modern distros are really very similar. The other big difference lies in your choice of a Desktop Environment (or a Window Manager), but that's a topic for a different article.

One sentence description for all significant Linux distributions follows.

- Debian             -> major distribution using the .deb package format and apt package manager
  - Ubuntu           -> major family of commercial distributions by Canonical Ltd.
    - Linux Mint     -> major desktop distribution for end users, pretty beginner friendly
    - Pop! OS        -> desktop distribution shipped with System76 machines, preinstalled drivers etc.
  - LMDE             -> it's Linux Mint without Ubuntu
  - Kali Linux       -> major pentesting distribution with a lot of offensive security tools
  - Tails            -> amnesic distribution focused on privacy and anonymity with Tor deeply integrated
  - Whonix           -> virtualized distribution focused on anonymity that routes all traffic through Tor
  - Raspberry Pi OS  -> default distribution for Raspberry Pi devices
- RedHat             -> major distro family using .rpm format for package management
  - Fedora           -> testing ground for RHEL, latest and greatest, operated by the community
    - Qubes OS       -> operating system that provides security through isolation with xen virtualization
  - CentOS Stream    -> second stage of the RHEL testing ground, used to be common enterprise linux
  - RHEL             -> the industry standard enterprise linux solution
    - Alma Linux     -> compatible alternative to RHEL
    - Rocky Linux    -> another compatible alternative to RHEL
    - Oracle Linux   -> the enterprise linux solution endorsed by Oracle
  - openSUSE         -> European equivalent to RedHat which also offers enterprise options
    - SLES           -> SUSE Linux Enterprise Server
    - SLED           -> SUSE Linux Exterprise Desktop
- Arch Linux         -> community-driven, simple, lightweight, flexible, fast, power-users delight
  - Arch Linux ARM   -> Arch Linux for ARM architectures
  - Arch Linux 32    -> Arch Linux for 32-bit CPUs
  - Artix Linux      -> Arch Linux without systemd
  - Asahi Linux      -> Arch Linux on Apple silicon
  - BlackArch        -> Arch Linux with pentesting tools
  - EndeavourOS      -> Arch Linux for newbies
  - Manjaro          -> the Ubuntu of Arch Linux family
  - Parabola         -> fully FOSS and libre distribution
  - SteamOS          -> distribution exclusively for gaming with Steam, by Valve
- Slackware          -> the oldest distribution still in use, solid and reliable, but not very modern
- Gentoo             -> highly customizable source-based distribution
- NixOS              -> reproducible, declarative and reliable distribution for advanced users
- Alpine Linux       -> truly minimal, small, simple, secure, very fast, utilizing musl libc and busybox
  - postmarketOS     -> alternative operating system for mobile devices
- OpenWrt            -> small yet powerful distribution for routers, firewalls, and other embedded devices
- Android            -> major operating system for smartphones and other mobile devices

For beginners I would recommend: LMDE, Linux Mint, Fedora. Advanced users will probably have their own preference, but personally I can definitely recommend Arch Linux. On a server I would primarly use: Alpine Linux, Debian, or any of the RHEL-based distros.

If you're not using GNU/Linux already, you should definitely try it!

Upskill

2024-08-15T00:00:00+00:00

It's good to brush up on your computer-magician skills from time to time.

Platforms with programming challenges are great for this. You can quickly solve a short task to regain some confidence in your fundemantal knowledge. One such platform is Codewars, which lets you solve simple tasks in a variety of programming languages. Solutions are tested on the platform and sometimes are constrained in terms of execution time, although the testing isn't as through as you might expect for competitive programming. The second platform I like is Codeforces, which aggregates tasks from various competitions. That's where you can really test your knowledge of algorithms. Then there's Project Euler, a collection of challenges often relating to some mathematical concepts.

Capture The Flag is a different class of challenges. On CTFtime you can find a list of upcoming CTF competitions, archive of past events, and many useful writeups. You will never run out of new tasks to solve. I particularly like Cryptohack, a CTF platform filled with challenges around the topic of cryptography. For a more structured approach to CybSec training, HackTheBox is probably the best choice. They offer a huge library of virtual labs and learning environments to break in quite a realistic fashion.

Nonetheless, the best way to upskill is to take up a personal project and to see it through. I always favor this over courses and dull exercises. Not to say that courses are bad, but I believe books and official documentation are the best learning resources, especially effective with practice.

Learning new programming languages or different technologies can also be very refreshing. Picking up a foreign language is great too. When you're tired of tech it might be a good idea to delve into literature from other scientific fields and hopefully come back with a new perspective. There's so many interesting topics to study, it's impossible not to get drawn into something when walking through a library.

Never stop learning!

New installations

2024-04-29T00:00:00+00:00

Installation guide on ArchWiki describes basics in more details. This list is just an overview, in practice the basic installation is mostly covered by 'archinstall'. In the past I wrote a post about partitioning for fully encrypted installations, check it out first if that's what you're aiming for.

partition the hard drive
format partitions
mount all the partitions
$ pacstrap -ci /mnt base
$ genfstab -U /mnt >> /mnt/etc/fstab
arch-chroot into the new installation
install kernel, run $ mkinitcpio -P
set up a bootloader like grub or systemd-boot
install additional packages: vim tmux networkmanager
set up a user account with sudo privileges
reboot

At this point you should have a basic working system installation, but there is a lot more to do.

set hostname, locales, keymaps, timezone
install additional drivers (microcode, nvidia, r8168) and load kernel modules you might need
regenerate mirrorlist with reflector
set up a desktop manager (lightdm) with a greeter (lightdm-gtk-greeter)
install a desktop environment (cinnamon) or a window manager (i3, sway)
install some AUR helper (paru, yay)
install your preferred set of packages
enable your preferred set of services
add your user to preferred groups
set up a firewall (ufw, gufw)
restore your dotfiles and custom configurations
change themes (mint-l-theme) and icons (papirus-icon-theme)
tweak your desktop environment settings just right
change your input method to fcitx with mozc
disable actions on lid close (/etc/systemd/logind.conf) and automatic suspend
install vim-plug and vim plugins
monitor filesystem integrity with AIDE

For some of these topics you can find helpful info on my blog.

Here's the reflector command that you can use for refreshing your mirrorlist:

$ reflector --verbose --latest 100 --fastest 30 --protocol https --sort rate --save /etc/pacman.d/mirrorlist

As a bonus, here's my lightdm-gtk-greeter.conf. I couldn't decide where else to store it.

[greeter]
background = /usr/share/backgrounds/lain-on-black.png
user-background = false
theme-name = Mint-L-Dark-Red
icon-theme-name = Papirus-Dark
font-name = Liberation Sans 12
indicators = ~power;~spacer;~separator;~clock;~separator;~spacer;~session
clock-format = %H:%M:%S - %a %d %h %Y
position = 5%,start -5%,end
a11y-states = ~contrast
hide-user-image = true

Happy hacking!

LLMs

2023-11-27T00:00:00+00:00

Large Language Models popped off with the release of ChatGPT, a service that launched almost exactly a year ago as a research preview at the end of November 2022 and quickly took internet by the storm. It gained over a million users in just a few days, and reached over a 100 mln users in a few weeks. They've caused a lot of fuzz which definitely accelerated the evolution of AI, and we've seen a lot of developments in that space over just this last year. While there is also a lot of valid criticism regarding AI, and plenty that I could pour out in that direction myself, LLMs sure are a damn useful tool. In simple terms, LLM provides text completions by predicting what the next word should be based on what it has seen previously - in mainly the context given, but also the training data. It would be pretty cool if we could run our own local instance of some open-source LLM, right?

Unfortunately there isn't much open-source in OpenAI's LLM solution like ChatGPT, but it will still serve a helpful example to better understand how it all works. With ChatGPT we use a web frontend for interaction with the language model like GPT-3.5 or GPT-4. You can also interact with available models through the API. GPT-3.5 itself is "a series of models" that consists of: 'code-davinci-002' (base model), 'text-davinci-002' (InstructGPT model based on the previous one), 'text-davinci-003' (an improvement over the previous one), 'gpt-3.5-turbo' (another improvement, optimized for chat). InstructGPT and other chat models are useful because of the fine-tuning towards instruction following and proper AI alignment. After the model has been trained for valid completions, there's an additional training step to teach it the pattern for interaction - that step is called fine-tuning. With open-source LLMs people can fine-tune the model to their specific need. The AI alignment is a preprogrammed bit of context that tells the model how to behave. In case of ChatGPT it is the invisible prompt that tells the model to play the role of a helpful chat assistant. All of this together makes LLMs useful for various tasks besides just text completions (which was needed for that software to take over mainstream). Without alignment the base GPT will not provide a dialogue by itself, but could be used successfully for code completion tasks. Below the interface, whether WebUI or API, there is a plethora of models to choose from, and that's the most interesting part. The "deprecations" page in OpenAI docs lists a whole graveyard of GPT-3 models with the naming convention of "[class]-{ada,babbage,curie,davinci}-[suffix]". They can all behave slightly differently. Following OpenAI, other tech giants such as G00gle, Micro$$oft, and Faceb00k have been toying with this tech too. A particularly interesting model was LLaMA by Meta AI (Faceb00k).

Open-sourcing LLMs started off with LLaMA, which was leaked on 4chan soon after release. Source code for inference was already published, but the leak contained model's weights (the result of model learning) which allowed the community to skip the access application process with the corp. It was easier for people to run it on their own machines and to fine-tune the model further. The significantly smaller 13B parameter model of LLaMA was exceeding the 175B parameter model of GPT-3 in performance. As with all things open-source, advancements were quickly made by the community. This laid ground for the de-facto standard platform of open-source LLMs. Llama2 was released a few months later, you can learn all about it here: https://ai.meta.com/llama/get-started/

This is how Facebook itself praises perks of open-source:

Crowd sourced optimization: The open source community has really embraced our models. To date, the community has fine-tuned and released over 7,000 derivatives on Hugging Face. On average, across standard benchmarks, these have improved performance on common benchmarks by nearly 10% with remarkable improvements of up to 46% for benchmark datasets like TruthQA.
Developer community: There are now over 7,000 projects on GitHub built on or mentioning Llama. New tools, deployment libraries, methods for model evaluation, and even “tiny” versions of Llama are being developed to bring Llama to edge devices and mobile platforms. Additionally, the community has expanded Llama to support larger context windows, added support for additional languages, and so much more.
Hardware support: The hardware community has fully embraced Llama as a key model architecture. Major hardware platforms AMD, Intel, Nvidia, and Google have boosted the performance of Llama 2 through hardware and software optimizations.

One of the platforms mentioned above is Hugging Face, which hosts a huge library of different models and datasets. Some of the models that I think might be worth checking out are: Mistral, Vicuna, WizardLM. HuggingFace also has a model leaderboard, so you can compare different open-source LLMs and pick something out by yourself: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

Besides fine-tuning the model and feeding it a good prompt, there there are still some parameters that you can change when you run the model. Check out this article for an idea: https://rentry.org/llm-settings

When you do finally run the model, really the most important thing to make good use of it is feeding it effective prompts. The model will respond better or worse depending on the way you express yourself. This sparked interest in "prompt engineering" - figuring out how to communicate with ready-made models effectively to make them do the right job and achieve desired outcomes. This could be very useful to you as an operator.

So, finally, how do you actually run this stuff?

With something like ollama or LocalAI, it's pretty simple and painless. You can easily interact with the LLM from your terminal or the HTTP API that is usually similar to the one from OpenAI (so maybe check out their docs too?). Both of these open-source projects have decent documentation.

From the website of LocalAI:

LocalAI acts as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families that are compatible with the ggml format. Does not require GPU.

An example of installing and running 'ollama' with 'mistral' model:

# pacman -S ollama
$ ollama serve &
$ ollama pull mistral
$ ollama run mistral

You can also install a local WebUI and plug into your locally deployed API. There are plenty open-source options to choose from, for example big-AGI or Cheshire Cat.

Where to go next? Here are two deligthful pieces with a ton of links to other sources:

No AI was harmed in the making of this blog post.

VPN security

2023-10-20T00:00:00+00:00

VPN stands for a Virtual Private Network and it's purpose is to connect together computers that are in different physical networks. This allows them to access and share resources as if they were all in the same LAN and even reach devices behind NAT. With proper routing they can also provide an exit to the internet somewhere else from the original point of connection and therefore serve as a proxy for connected devices.

The proxy feature was not the point of a VPN, but because of heavy advertising online, many people started to see it that way. There are a few problems with using commercial VPNs as a proxy for the sake of security; I'll go through them briefly.

Commercial VPN providers claim it is more secure to use a VPN, because then your connection is encrypted. The truth is, if you're using HTTPS (and you most likely are - see the lock icon in the URL bar), then your connection is encrypted by default anyway. As long as the Certificate Authority (CA) is not compromised, the connection between you and the website is private. People on your network will not be able to "steal your passwords" easily. They might be able to see what you connect to, what website you are talking to, but they will not see what you "talk about" with the online service. Snooping on the traffic used to be possible back in the plain HTTP days, but now it's only viable if your machine trusts a compromised CA. On your work machine that could be the case: it might be set up to trust a CA of your employer so they would be able to see what you're doing. When you click on the padlock in the url bar you can see details that will tell you who the CA is, so you can know whether certificates in use were compromised by your employer. VPN will not help you if the CA is compromised for real by someone malicious. The advertisement claim is about MITM attacks in an outdated scenario and for the criminal it won't work out as easily as they claim anymore. Yes, a VPN will make a MITM attack harder, but MITM in of itself is not really that useful, and definitely not as powerful as they claim, thanks to the common use of encryption nowadays. Check out my post on ARP to see an example of a MITM attack.

As I said, people on the same network might not see what you're talking about, but they can see what server you're trying to talk to. When you connect to a VPN you only talk to one server and the rest of your traffic goes out to the internet from there. People on your network might not see who you're talking to besides the VPN server, but the VPN server does! And everyone still knows what VPN server you were connected to! If this is a commercial VPN, the provider most likely has a KYC (know your client) policy that makes it way easier to indentify you than if you were simply connecting to the endpoint directly. They might claim to never store logs, but: someone might still monitor that traffic, perhaps collect it and analyze it in a different way, exit nodes might be under heavier surveilance than your regular connection, with servers under foreign jurisdiction you can't always be sure what is going on there and how much the provider can or can't tell you about it. At the end of the day, you don't know what's going on with your traffic at the provider's side and there is no way for them to prove what is really happening with it. The only way to ensure security and privacy is through encryption end-to-end. For anonymity VPN is not a very good idea. The key question is: who do you trust more - your ISP or the VPN provider? Maybe in your case it makes sense to access the internet with a connection originating from a different country. Commercial VPN advertisements often mention bypassing geoblocks. If that is your goal, you might want to read more about Tor or check out my previous post on piracy.

Commercial VPN providers also claim to hide your IP address and thus make you anonymous. That's not how it works. Anonymity is more about how you use the service than what exact service you use. If you want to be more anonymous you should use Tor instead of commercial VPNs. The VPN provider can most likely identify you already. They know the account you use to connect, they can see all the details such as where you connect from, when, for how long, the details of your payment method, your browsing patterns, perhaps your web browser fingerprint, and perhaps a lot more details in case they have a stronger KYC policy. That's not even the biggest issue here. When you're using the VPN as a proxy with the same IP address during the whole session - everything you do during that session is still interconnected. If you deanonymize yourself by logging into any account, you deanonymize the whole session. Tracking by IP is also not the only option for anonymity adversaries. Devices we use and the software we choose can also be used to fingerprint and track us, even between different connections and websites, while being much more reliable and precise. Let's just say that anonymity and online tracking is a wide topic. Your anonymity will crumble if you manage your online identities wrong and no VPN can help with that. Ideally you would like to have a common fingerprint and use a different exit node for each connection. That's what Tor aims to accomplish. It's what it does for free.

In summary, commercial VPN gives a false sense of security and you more than likely do not need it. Nevertheless, you might want to set up a VPN for a lot of different reasons, but not necesarilly for the sake of "security". In future post I might explain how to do it. There are many VPN technologies to choose from.

PPTP - unsecure 90s protocol spawned by Micro$$oft, avoid this one
L2TP - another 90s protocol, to be used in tandem with IPsec, be careful with this one
OpenVPN - popular open-source solution, both protocol and server software, requires additional client software, but everything about it is fairly easy to set up and works great
IKEv2 - strongSwan, Openswan, Libreswan - family of FOSS servers implementing IPsec, IKEv2, very cool
SSTP - another protocol spawned by Micro$$oft, supported OOTB on Window$$, uses SSL/TLS on port 443 by default
WireGuard - modern, fast and simple, secure protocol, included in the Linux kernel, but with wide support for all the platforms. If you're wondering what you should use, this is what you should use.

There are some commercial VPN server solutions too, but I am convinced that WireGuard trumps them all. If you want to deploy a server quick and easy then OpenVPN is also a pretty good choice.

Piracy

2023-09-10T00:00:00+00:00

0. PIRACY IS GOOD

I hope if you're reading this you already understand that.

1. BLOCK ADS

This is no joke, this is extremely common vector of attack that gets people infected with malware or fall for phishing and lose access to their accounts. Also, ads are annoying. Fuck advertisers.

Use the best adblocker there is: https://ublockorigin.com/

Find the uBlock Origin extension for your browser and install it (that's two clicks to be much safer online). The website above will help. You can use it both on Chrome and Firefox, but ad-blocking generally works better on Firefox. Google is a business selling ads, obviously it's in their incentive for ad-blockers to be less effective.

2. SEARCH AND YOU SHALL FIND

Again, searching on Google will not be very effective, since their search engine is heavily censored. You know, to comply with demands of industry giants that advertise there and spend big money. Also DMCA.

A meta search engine like SearXNG would be more effective, but it's a little more advanced. Find a working instance and search from there: https://searx.space/

DuckDuckGo should give decent results for casual searches and it's even easier to use than Google, since it doesn't include ad spam in top results: https://duckduckgo.com/

3. STREAMING

Now that you have WORKING AD BLOCKING set up and you know how to effectively search the web, it's time to finally watch something.

Here's a list of a few websites you can use for streaming movies and series. Very simple to use. Just be cautious. Don't fall for scams and don't fall for phishing. Exact domains can change with time as they get shut down, but at least you know what to look for.

https://fmovies.to/ - favourite for getting movies easy and quick
https://susflix.tv/
https://gogoanime.tv/

There are lists of links too:

https://www.reddit.com/r/FREEMEDIAHECKYEAH/wiki/index/ - Piracy Wiki on Reddit
https://www.reddit.com/r/FREEMEDIAHECKYEAH/wiki/video/ - list of video streaming sites from that Wiki
https://shakil-shahadat.github.io/awesome-piracy/ - huge list of resources and websites related to piracy

If you choose to create an account on any of the piracy websites - don't use anything that could be related to you. Use a completely random password or some simple throw-away that you've never used anywhere before. You can be identified by the password you use, not only by e-mail or username. Don't reuse passwords.

For music you could use a modded Spotify app for free or download your music from elsewhere. Again, see the resources above. If you find the modded .apk I would advise to scan it on virustotal.com before installing it on your phone.

Don't ever pay for streaming.

4. DOWNLOADING

You can download anything you can stream and more. When you're streaming something you're actually downloading the media piece by piece while displaying each piece as it is ready to be played. If you save these pieces when you recieve them - you can effectively just download the media file. Some of the streaming sites mentioned above will give you an option of direct download for simplicity. For others, like YouTube, you may have to use a specific downloader, for example: yt-dlp.

To download a YouTube video:

$ yt-dlp [link]

To download audio only:

$ yt-dlp -x [link]

You can find direct download links to most things with enough effort.

https://chomikuj.pl/ - Polish file storage and file sharing website. You pay a little bit for the data transfer, but there is a lot you can find there, especially if you're looking for something in Polish language.
https://sci-hub.st/ - Scientific papers unlocked.
https://libgen.rs/ - Library Genesis is a huge collection that contains non-fiction books, fiction, scientific articles, and overall a lot of useful stuff. Their files are also available through torrent download and on IPFS.
https://annas-archive.org/ - Amazing website that allows you to search through a vast library including Sci-Hub, LibGen and Z-Library resources.

4b. TORRENTS

The rest of this section will be all about torrents.

There's one simple catch when it comes to torrents - it's a peer-to-peer network, which means it only works when people who download also upload the same content for others to use. Uploading is where the copyright infringement lies. In short, downloading should be legal, uploading may not be. In your torrent client settings you can set limits for upload or use a proxy/vpn to hide yourself a little bit; although in some countries, as long as you don't pirate movies produced in that country, you're probably better off just using your raw connection without a foreign VPN.

You need a torrent client to open .torrent files or open "magnet links" to download stuff.

My favorite torrent client is Transmission: https://transmissionbt.com/

You will need trackers to track the peers that seed (upload) the file you want to get. The list of trackers that your torrent client will use is usually included with the .torrent file or magnet link. There are private trackers that may provide better service or have some rare content available, but I won't go into details on this.

Here's a list of a few websites where you can get your magnet links:

https://ext.to/ - torrent aggregator with multiple sources
https://thepiratebay.org/ - a classic, giant of the internet history, one and only, The Pirate Bay
https://1337x.to/ - good source for movies and series
https://nyaa.si/ - best quality anime and all that nerd weeb shit

When I say "best quality anime" I actually mean it. It's usually better than any streaming. The highest quality, best encoded, best looking releases you can find are over there. You're welcome.

5. STAY INFORMED

News website that covers piracy-related topics and more: https://torrentfreak.com/

Common Lisp resources

2023-08-12T00:00:00+00:00

Common Lisp is a lisp dialect with an official ANSI specification (X3J13), but multiple different implementations. The one I primarly use is ECL, but I'm also using SBCL sometimes, which is the most popular choice. There should be no difference regardless of which implementation you go with.

Lisp is a wonderful language, and below is a list of resources for learning more about Common Lisp, also for my future reference.

Common Lisp HyperSpec:

The Common Lisp Cookbook:

https://lispcookbook.github.io/cl-cookbook/

Common Lisp Style Guide:

https://lisp-lang.org/style-guide/

Quickdocs:

https://quickdocs.org/

Quickref, index of libraries:

https://quickref.common-lisp.net/index-per-library.html

Book "Practical Common Lisp" by Peter Seibel:

https://gigamonkeys.com/book/

Book "Common Lisp the Language, 2nd Edition" by Guy L. Steele Jr.:

https://www.cs.cmu.edu/afs/cs.cmu.edu/project/ai-repository/ai/html/cltl/clm/node1.html

Book "On Lisp" by Paul Graham:

http://www.paulgraham.com/onlisp.html

Book "Let Over Lambda" by Douge Hoyte:

https://letoverlambda.com/

samba

2022-09-29T00:00:00+00:00

All things micro$$oft suck ass. AD and SMB are no exceptions.

Active Directory (AD) is "a directory service for Window$$ domain networks", essentially used for identity management. You can think of it as a Window$$-centric authentication and authorization service. SMB is a file sharing protocol pushed out with Window$$ OS. Samba is the interoperability suite to utilize this insanity on some sane OS.

As per usual, more info on ArchWiki:

https://wiki.archlinux.org/title/Samba

Extensive docs are out there, too:

https://www.samba.org/samba/docs/Samba-HOWTO-Collection.pdf

If all you need to do is access files over SMB, your best bet is to use gvfs-smb and save yourself a headache of trying to figure out how it all works. Using GVfs allows you to connect and nicely browse shares with file managers like Nemo or Nautilus. You have to know the IP address or DNS name of your share. Simply type it in with "smb://" protocol prefix as a location in Nemo and you'll be prompted for credentials to access the share. SMB also allows for sharing printers somehow, but I've never tried it.

If you need to play with AD a little bit more, read on. Then refer to this ArchWiki page:

https://wiki.archlinux.org/title/Active_Directory_integration

A few configuration details you'd like to note down: NetBIOS domain name, DNS domain name, Kerberos realm, DC1 (hostname/IP), DC2, etc.

To set up your DNS - change /etc/resolv.conf according to the template:

nameserver [DC1 IP]
nameserver [DC2 IP]
search [internal.domain.tld]

Test it with:

$ nslookup -type=SRV _kerberos._tcp.internal.domain.tld.
$ nslookup -type=SRV _ldap._tcp.internal.domain.tld.
$ nslookup dc1.internal.domain.tld.

For Kerberos it may be neccessary to modify /etc/krb5.conf if authentication doesn't work properly.

Configuring samba cosists of a few steps, but I won't go into details: basic config, disabling printers, UID/GID mapping config, NSS config for UID/GID mapping. You may need to run these services: 'smb.service', 'nmb.service', (optional) 'winbind.service'. Again, consult references linked above for more information.

To join a domain try:

$ net ads join -U [username]

or:

$ samba-tool domain join [internal.domain.tld] MEMBER -U [username]

A few commands for testing:

$ net ads info
$ net ads lookup
$ net ads status -U [username]

$ wbinfo -u
$ wbinfo -g
$ wbinfo --ping-dc

Now some more SMB commands...

In case you didn't like gvfs-smb that I suggested, you may have shot at using smbclient.

Listing:

$ smbclient -L "\\12.23.34.45" -U [username]

Connecting:

$ smbclient "//12.23.34.45/DIRECTORY" -U [username]

I honestly don't know about the slashes here. Micro$$oft is always going backwards so they prefer backslashes. Anything else would escape on a backslash. How does it mix in your shell? IDK, just try it I guess? ¯\_(ツ)_/¯

SELinux

2022-07-25T00:00:00+00:00

SELinux is a Linux Security Module (LSM). When the kernel does something like a permission check and it passes OK, the security module is queried right after to provide additional security checks. You will find SELinux on RedHat based distributions and Android. Another popular option is AppArmor, used for example in Ubuntu. There can only be one LSM is use at a time, so you won't ever use both of them at once.

SELinux provides additional context associated with certain paths to provide an extended permission model.

Directory listing with SELinux context:

$ ls -laRZ .

List possible context values:

$ semanage fcontext -l

Refresh assigned context values:

$ restorecon -Rv .

Change context:

$ chcon

To automatically refresh context values and relabel the system on reboot:

$ touch /.autorelabel

arduino-cli

2022-05-28T00:00:00+00:00

Arduino CLI is pretty convenient and definitely better than the default Arduino IDE.

Documentation is here: https://arduino.github.io/arduino-cli/

Generate configuration:

$ arduino-cli config init

Add custom boards to arduino-cli.yaml by modifying the additional_urls list:

board_manager:
  additional_urls: ["http://digistump.com/package_digistump_index.json"]

Update index and install the core:

$ arduino-cli core update-index
$ arduino-cli core search digistump
$ arduino-cli core install digistump:avr

Identify your Fully Qualified Board Name:

$ arduino-cli board listall

Compile and upload:

$ arduino-cli compile -b digistump:avr:digispark-tiny sth.ino
$ arduino-cli upload -b digistump:avr:digispark-tiny sth.ino

For the upload to work with digispark-tiny board you will need to install micronucleus bootloader first.

$ yay -S micronucleus

In case you need it, here's a digispark troubleshooting page: http://digistump.com/wiki/digispark/tutorials/linuxtroubleshooting

AIDE

2022-05-05T00:00:00+00:00

AIDE stands for Advanced Intrusion Detection Environment and it keeps track of system integrity by holding a database of file details. This can help you catch unauthorized file modifications and other tampering attempts which could be caused by malware, misbehaving software or bad actors on your server.

The weak point of this solution is the dependence on trusted AIDE database. It is created with the initial scan, so if your system has been infected beforehand - AIDE will be of no help.

The database should be updated periodically to keep single check runs nice and clean. Keeping track of the AIDE database integrity is on the administrator. It is still much easier to track integrity of only one file by hand instead of the whole system, so AIDE turns out to be quite useful nonetheless.

On Debian the default main configuration is located at /etc/aide/aide.conf with more configuration files under /etc/aide/aide.conf.d/. Database locations are defined in the main configuration file. These paths are not baked into the binary, so you have to pass a '--config' parameter when using AIDE on Debian.

On Arch Linux the default configuration is located at /etc/aide.conf and the path is automatically recognized. You can confirm default locations with:

$ aide --version

If you're using Arch Linux instead of Debian, simply omit the --config parameter in the commands below.

To initiate the database:

$ aide --config /etc/aide/aide.conf --init

This will perform initial scan and drop a new database file. Updates work similarly, but they will produce a longer report, because they also run checks.

Every time you update the database you have to manually replace the old version with the new one. For Debian:

$ mv /var/lib/aide/aide.db{.new,}

To perform the update:

$ aide --config /etc/aide/aide.conf --update

To run a check you can issue:

$ aide --config /etc/aide/aide.conf --check | tee aide.out

The output might be longer than you think, but you can inspect the aide.out file later.

Every action outputs multiple hashes of your database. AIDE will check the integrity of your files, but the database it uses must be trusted. Makes sure these checksums don't change unexpectedly, as it is crucial for the integrity.

CBC bitflip

2021-12-30T00:00:00+00:00

Advanced Encryption Standard (AES) is a well-known encryption standard and a common name for the symmetric block cipher called Rijndael - originally named after the surnames of researchers who proposed it: Joan Daemen and Vincent Rijmen.

The core encryption algorithm can be applied to blocks of 128 bits with a key of 128, 192, 256 bits. Besides the core encryption algorithm there are multiple modes of operation, which define how to apply the algorithm to multiple blocks. Let's have a quick look at two very basic ones.

Electronic Code Book (ECB) mode is the simplest. Plaintext is split into blocks and each one of them is encrypted in the same way. "ECB Penguin" image shows very well why this is not ideal. Two blocks of plaintext that are the same will give the same output. If we can control some part of the input and we can see the encryption output, we can recover parts of the plaintext without knowing the key. By carefully aligning blocks we can guess and match the encrypted part character by character.

Cipher Block Chaining (CBC) mode is only slightly more complicated. In this mode each block of plaintext is XOR-ed with previous ciphertext before encryption. IV is the initialization vector. The scheme looks something like this:

(P_1 ^ IV)  ---[block encryption]---> C_1
(P_2 ^ C_1) ---[block encryption]---> C_2
(P_3 ^ C_2) ---[block encryption]---> C_3
...
(P_n ^ C_(n-1)) ---[block encryption]---> C_n

This has an interesting property. Again, if we control some part of the input and we can see the encryption output, we can manipulate the decryption output by modifying the ciphertext without knowing the key.

Let M be the malicious input.

encryption:
(P_1 ^ IV)  ---[block encryption]---> C_1
(P_2 ^ C_1) ---[block encryption]---> C_2
(P_3 ^ C_2) ---[block encryption]---> C_3
(P_4 ^ C_3) ---[block encryption]---> C_4

modification:
C_2 ^ M = C_M

ciphertext:
[C_1, C_M, C_3, C_4]

decryption:
C_1 ---[block decryption]---> (P_1 ^ IV) ---[^ IV]---> P_1
C_M ---[block decryption]---> (block of random bullshit)
C_3 ---[block decryption]---> (P_3 ^ C_2) ---[^ C_M]--->
 (P_3 ^ C_2 ^ C_M) = (P_3 ^ C_2 ^ (C_2 ^ M)) = P_3 ^ M = P_M
C_4 ---[block decryption]---> (P_4 ^ C_3) ---[^ C_3]---> P_4

We have modified some block of plaintext.

If we can also control the IV we can fix up the blocks we've broken on the way. Oftentimes we don't care though. You may hit some unfortunate bytes once the block of garbage decrypts, but sometimes it's enough to resolve it by messing with the block some more instead of fixing up the decryption output all the way.

This attack is pretty simple in principle, the harder part is to find where exactly to apply it.

I'll just hint that there are more attacks on this mode of operation. An interesting example is a padding oracle attack, but that's a topic for another post.

Partitioning

2021-10-11T00:00:00+00:00

Partitioning is the most tedious part of the OS installation process. Everything else is either very fast to set up or can easily be automated. Even with declarative systems, they mostly assume you already have prepared your hard drives before applying the configuration. Partitioning cannot be mutated as easily as configuration, therefore it requires proper planning and a lot of attention. I tend to think about it more than I should.

Needless to say, I want my stuff encrypted. My basic setup follows this article:

https://wiki.archlinux.org/title/Dm-crypt/Encrypting_an_entire_system#LVM_on_LUKS

With coreboot (or libreboot) I can encrypt the entire drive without leaving the boot partition readable. My GRUB payload has LUKS and LVM support compiled in. It can decrypt the drive and boot straight away or load another GRUB config file from the decrypted drive.

Check out guides on Libreboot's website before commit 48b04d0993 to see how that's done:

https://notabug.org/libreboot/lbwww/src/8844c201ef0d1ab856fed2aa5148b89100fffe0d/site/docs/gnulinux

The parabola one was especially helpful when I was first setting it up. Currently it is available here:

https://wiki.parabola.nu/Installing_Parabola_on_Libreboot_with_full_disk_encryption_(including_/boot)

Unfortunately not all my machines are officially supported by the coreboot project. In such a case the UEFI boot must suffice. That usually means leaving out an unencrypted EFI boot partition which makes the GRUB config and possibly the kernel vulnerable to tampering. Unless we also properly set up Secure Boot... I won't.

Just be wary - when someone gains physical access to the machine, although they won't be able to read the valuable content of the drive right away, they could still do a lot of harm. If that happens to you - do not boot from that drive again. Boot from some other safe media, nuke all readable content on the drive, then regenerate it from within the system.

We will use fdisk cli for partitioning, however if you prefer a graphical utility - GParted is a nice one.

$ fdisk /dev/sda

First we choose "g" to create new GPT partition label, then "n" to create a new partition. The boot partition should span a few hundred MB (300M should be enough). Use "t" to change it's type to "EFI System" - option 1. Next is the general partition for LUKS which should cover all available space left on the drive. Write changes with "w".

Format the boot partition with:

$ mkfs.fat -F32 /dev/sdxY

Format the general LUKS partition with:

$ cryptsetup -v --iter-time 600 --type luks1 luksFormat /dev/sdXY

The type flag was needed for GRUB compatibility. If your bootloader is fresh and shiny you may skip it. The iteration time is set to 2 seconds by default. I prefer to set it lower, like 600ms in the example above.

Options and defaults are described here:

https://wiki.archlinux.org/title/Dm-crypt/Device_encryption#Encryption_options_for_LUKS_mode

Basic LVM setup:

$ cryptsetup luksOpen /dev/sdxY cryptlvm
$ pvcreate /dev/mapper/cryptlvm
$ vgcreate vgrp /dev/mapper/cryptlvm
$ lvcreate -l 100%FREE vgrp -n root
$ mkfs.ext4 /dev/vgrp/root

At this point you can mount your partitions and install the system or whatever. In the end we get to something like this:

drive -> LUKS -> LVM -> ext4

We need to tell our kernel how to deal with this. Therefore in /etc/mkinitcpio.conf we set:

HOOKS=(base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 filesystems fsck)

Run mkinitcpio:

$ mkinitcpio -P

We also need to set kernel parameters:

 cryptdevice=UUID=[device-UUID]:cryptlvm root=/dev/vgrp/root

How to do that depends on a bootloader. For GRUB we can simply modify grub.cfg (on the boot partition or in cbfs). For systemd-boot we would put those parameters on the "options" line of a proper /boot/loader/entries/*.conf file.

Of course we can use something else instead of the good old ext4 filesystem. My first pick would be btrfs which offers some cool features like subvolumes and snapshots. I don't feel like fully explaining it now though (maybe in a btrfs post?). Long story short - subvolumes could suffice for partitions and the LVM layer may not be needed, since btrfs is flexible enough by itself.

For automatic snapshots on btrfs: Snapshot destination should be in a different subvolume than the directory you are snapshotting. I also highly recommend putting /home on a different subvolume or on a different partition altogether. Therefore you need at least 3 subvolumes: root, home, snapshots. Snapshot root and home separately. Check out snapper and snap-pac (both available in Arch Linux community repo).

Here's an example of kernel parameters for root on a btrfs subvolume:

 cryptdevice=[device-partition]:cryptroot root=/dev/mapper/cryptroot rootfstype=btrfs rootflags=subvol=/@

If you decrypt with coreboot payload, you may want to add a keyfile and additional kernel parameters. This should be described in at least one of the articles I linked at the top.

Some people also like to partition off swap space, although you can simply use a swapfile. This is not necessary, but if you'll want to hibernate your system or you've ever run out of RAM - look into setting up swap next.

epnel algo

2021-10-05T00:00:00+00:00

Recently I had a technical interview for a junior positon with a following task: given a string, find the length of the longest substring without repeating characters. Fairly simple, but due to stress I wasn't able to properly solve it and completely failed the interview as a result. The problem and it's solutions are described here: https://www.geeksforgeeks.org/length-of-the-longest-substring-without-repeating-characters/

To make up for that epic fail I decided to implement the solution in my own esoteric language that I wrote sometime in high-school - epnel. Below is a valid Python solution, adapted for easy translation.

inputs = ["abcabcbbc", "abcabcbbacbd", "ddabcdedd"]

for x in inputs:
	d, b, r = dict(), 0, 0
	for i, c in enumerate(x):
		if c in d:
			nb = d[c] + 1
			if nb > b: b = nb
		d[c] = i
		nr = i - b + 1
		if nr > r: r = nr
	print(x, r)

You can find the epnel language on my GitLab: https://gitlab.com/wykwit/epnel

The curiosity that is epnel was also written in Python, but it doesn't provide much IO functionality. For that purpose, we can use the host language again.

import epnel

inputs = ["abcabcbbc", "abcabcbbacbd", "ddabcdedd"]

# convert input string to an integer for epnel to handle
convert_input = lambda x: int.from_bytes(bytes(x, "ascii"), "big")

# main epnel code
algorithm = """
= 0 {}
"""

# apply algorithm to each input
for x in inputs:
	s = algorithm.format(convert_input(x))
	p = epnel.Parser(string=s)
	# and because we want to have a clean start
	epnel.interpreter.symbol_table.clear()
	r = epnel.interpret(p.root())
	print(x, r)

Mind that it doesn't matter whether we traverse the string from the beggining or from the end - we should get the same result, therefore endianness in our conversion shouldn't matter. What we need now is an actual algorithm to plug into the code above. Let's start off with a simple example and translate the loop first.

All variables (previously undefined symbols) have a value of zero by default. Our sample input string translates to 1796423792463439880801, we can set it like this:

. in 1796423792463439880801

There are no indentation rules in epnel. Sometimes I myself am not sure what would make most sense. The only requirement is that all parameters should be separated with at least a single whitespace, everything else is up to the programmer. I try to make my examples clean, but it's perfectly fine to simply have everything in a single line.

Once we have a loop, we would want that loop to execute some body. For now we can simply count the number of iterations to make sure our loop works right.

, body . r + r 1 !

Now the actual loop would look something like this:

, loop
  ? > in 0
  | . c & in 255
  | body
  | . i + i 1
  | . in / in 256
  | loop !
  !
  loop

We define a loop and call it in the last parameter of the definition. First statement we execue is a check on whether there is still some input to handle. This condition will end recursion. Then we assign current character to a variable and execute our body. Before we recursively call the loop again, we increment our loop counter and remove the character we've just looked at. Finally we finish the loop off with a couple of exclamation marks to fill remaining parameters with boolean false.

We can chain these three blocks with ampersands (binary and) and wrap with another binary or to get a clean result.

| & . in 1796423792463439880801
  & , body . r + r 1 !
  & , loop
      ? > in 0
      | . c & in 255
      | body
      | . i + i 1
      | . in / in 256
      | loop !
      !
    loop
  !
r

Save the file as loop.e and we get a following result when we run it.

$ epnel loop.e 
9

Our input string consists of 9 characters, so this matches our expectations. Apparently the loop works.

We can now move on to translate the loop's body, which consists of a conditional statement and a few assignments. This is simple enough, but in our reference we are using a dictionary (hashmap) which doesn't have an obvious translation to epnel. Since we deal with characters that fit in one byte, we don't really need a dictionary (hashmap) - an array should suffice. Still, how do we solve this in epnel? We can use regular assignments and the (most confusing) epnel dereference operator. Assignments in epnel could overwrite numeric values and additionally confuse some code (which is awesome btw). Just to stay on the safe side, we can denote a variable to be our offset into negative numbers, which are not used in the algorithm.

For more details on assignments check out README in the epnel repo. Dereference is called "number reassignment". Play around with the REPL a little to figure out how exactly it works.

In the end - the body translates neatly.

, body
  | ? = : + d c 0
    ! 
    | . nb + : + d c 1
      ? > nb b . b nb !
  | . + d c i
  | | . nr + - i b 1
      ? > nr r . r nr !
  !
!

The final solution takes way more than 10 LOC and is more complex in practice, but theoreticaly the underlaying algorithm is also linear (and it looks cool).

import epnel

inputs = ["abcabcbbc", "abcabcbbacbd", "ddabcdedd"]

# convert input string to an integer for epnel to handle
convert_input = lambda x: int.from_bytes(bytes(x, "ascii"), "big")

# main epnel code
algorithm = """
| & | . in {}
    | . d - ! 9999
    !
  & , body
      | ? = : + d c 0
        ! 
        | . nb + : + d c 1
          ? > nb b . b nb !
      | . + d c i
      | | . nr + - i b 1
          ? > nr r . r nr !
      !
    !
  & , loop
      ? > in 0
      | . c & in 255
      | body
      | . i + i 1
      | . in / in 256
      | loop !
      !
    loop
  !
r
"""

# apply algorithm to each input
for x in inputs:
	s = algorithm.format(convert_input(x))
	p = epnel.Parser(string=s)
	# and because we want to have a clean start
	epnel.interpreter.symbol_table.clear()
	r = epnel.interpret(p.root())
	print("{} : {}".format(x, r))

It looks even more cool when you get rid of indendation and rename variables that are too descriptive.

:) | & . x 1796423792463439880801 & . d - ! 9999 & , l ? > x 0 | . c & x 255 | | ? = : + d c 0 ! | . _b + : + d c 1 ? > _b b . b _b ! | . + d c i | ? > + - i b 1 r . r + - i b 1 ! ! | . i + i 1 | . x / x 256 | l ! ! l ! r 
3

That's how problem-solving in epnel works. That is, if you can come up with the algorithm on time, of course.

email regex

2019-12-19T00:00:00+00:00

email regex - when you need to find email addresses in a long document or validate html forms

This was also one of Google Code-In tasks.

Basic regex syntax:

\s - matches a whitespace
() - matched group
[] - matches any range of characters in the brackets
[a-zA-Z0-9] – matches alphanumeric characters
[]+ - matches given range multiple times
(L|R) – a group with alternation, matches either left or right side expressed inside the parentheses
[]* - matches given range zero or more times
[]? - matches given range once or not at all

According to RFC5322 section 3.4.1 the e-mail address is specified as follows (in ABNF):

addr-spec      =    local-part "@" domain
local-part     =    dot-atom / quoted-string
domain         =    dot-atom / domain-literal
domain-literal =    [CFWS] "[" *([FWS] dtext) [FWS] "]" [CFWS]
dtext          =    %d33-90 / %d94-126 ; printable ASCII not including "[", "]", or "\"

The obsolete alternate parts were omitted.

dot-atom may not begin with a dot, it consists of:

any alphanumeric character or any of "!#$%&'*+-/=?^_`{|}~."

Domain name syntax is specified elsewhere, for simplicity let's assume only alphanumeric characters and dashes are allowed in domain-names. Domain names may also consist of subdomains, so we should match dots too. quoted-strings in a local-part of the email address and domain-literal in the domain part is rarely used, especially in web-forms etc.

In practice matching a following pattern should be sufficient (in EBNF):

email-address =    dot-atom, "@", domain
domain        =    alphanumeric, {["." | "-"], alphanumeric}, ".", alphanumeric, {alphanumeric}
dot-atom      =    (legal-char - "."), {legal-char}
legal-char    =    alphanumeric | ? any of "!#$%&'*+-/=?^_`{|}~." ?

Therefore a following regex should match most of e-mail address names in a practical scenario:

([#-'\/-9A-Z^-~!*+=?-][#-'--9A-Z^-~!*+=?]*@[a-zA-Z0-9]([.-]?[a-zA-Z0-9]+)*\.[a-zA-Z]+)

Some additional explanations regarding this regex:

@ - matches a literal "@"
\. - matches a literal "."
[a-zA-Z0-9-.]+ - matches any alphanumeric character, "." or "-" multiple times (to get the domain name and subdomain names)

The first part matches a dot-atom (any legal character excluding a dot followed by any number of legal characters). Parts are separated by a "@". Then the second part matches alphanumeric characters divided by dots or dashes. The string must end with alphanumeric top-level domain.

ARP

2019-12-18T00:00:00+00:00

ARP stands for Address Resolution Protocol. It's purpose is to resolve IP addresses in local networks (LAN) to MAC addresses used for direct communication. Every network card has a unique MAC address, so it's convenient to know which hardware you are supposed to communicate with, when you're addressing some IP.

It's analogous to DNS, but on a lower level. DNS resolves domain names to IP addresses - it works both in LAN and in the Wild Internet. ARP resolves IP addresses to MAC addresses, but from what I know it is used in LAN only. It doesn't make much sense to use MAC addresses outside of LAN, because you can't access hardware that is not in the range of your antenna. Therefore IP's usually suffice.

If we want to mess with our LAN, it makes sense to mess with MAC addresses too. Computers usually resolve IP's on LAN by first looking through their local ARP table. If the target is not found, they send an ARP request to the broadcast address that looks somewhat like "Who has 192.168.50.69? Tell 192.168.50.143". Every computer on the network is subscribed to the broadcast, so then the holder of the IP would respond to us directly: "192.168.50.69 is at [MAC addr]". This is all we need to know to start a communication.

This is also how scanning for IP's on LAN works.

$ nmap -sP 192.168.1.1/24

When you issue this command, nmap floods your network with ARP requests for every possible IP in your specified range. Sooner or later someone will respond. Blocking ARP requests usually breaks TCP/IP connection, so there's a high chance victim machine would reply.

Let's see if we can somehow manipulate ARP. For that purpose we will use dsniff, which includes tools like arpspoof or dnsspoof. To demonstrate how this works I've MITM'ed my friend's laptop.

On the photo you can see my laptop sending out malicious packets and my friend's laptop displaying attacker-controlled website when they try to access a legitimate domain. Now how did we pull this off? It's actually very simple.

First we need to know the IP of our victim and our gateway.

$ ip a show dev wlp2s0

or on windows simply:

> ipconfig

We are on the same network, so we can execute the attack.

$ arpspoof -i wlp2s0 -t 10.14.75.17 -r 10.14.72.1

Where 10.14.75.17 is my friend's IP and 10.14.72.1 is the gateway.

It seems to work. The gateway thinks we are the victim computer and victim treats our computer as the gateway. We've fooled them. The connection now goes from the victim computer to our machine instead of the real gateway. This is called a man-in-the-middle (MITM) attack, because our machine is now in the middle of the connection between victim and the internet. The technique used is known as ARP spoofing or ARP poisoning, because we've spoofed some ARP requests to poison the ARP table of victim and gateway, and redirect their traffic through our machine.

Here's a nice graph from Wikipedia to visualise how the connection looks like during the attack.

For the sake of demonstration we set up a simple website.

$ vim index.html
$ python2 -m SimpleHTTPServer 80

And spoof DNS, so that example.com points to our IP address.

$ echo "10.14.75.206\texample.com" >> hosts
$ dnsspoof -i wlp2s0 -f hosts

Now when we go to example.com from the victim computer, we ask our gateway what is the IP of example.com. We act as the gateway, so we repond with our own IP address. The victim goes to our IP instead of the real one and sees our website.

As you can see, arpspoof attack is really powerfull. In the MITM scenario we can sniff and manipulate victim's traffic however we want. Hopefully you now understand the principle of how this attacks work.

This post was made for Google Code-In and screenshots were requested as a part of the task.

fcitx-mozc

2019-11-30T00:00:00+00:00

There is a difference between an input source (keyboard layout) and an input method. When you browse your DE settings, they only really let you change the input source. If you'd want to type in Japanese for example, even if you change your input source to JP it only changes the keyboard layout. You needn't mess with the settings in that regard. All you have to do is change your input method. Perhaps you already know: to type in Japanese, you need a "Japanese IME". Fcitx is an input method framework and Mozc is a Japanese Input Method Editor. Plug Mozc into Fcitx and you'll be good to go.

The ArchWiki article on fcitx is way too long: https://wiki.archlinux.org/index.php/Fcitx

And the article on Mozc refers to the previous one: https://wiki.archlinux.org/index.php/Mozc

The setup is not even that complicated, it just looks like a lot of work. Here's what you really have to do:

$ pacman -S fcitx-im fcitx-mozc fcitx-configtool
$ cp /etc/xdg/autostart/fcitx-autostart.desktop ~/.config/autostart/
$ gtk-query-immodules-3.0 --update-cache
$ gtk-query-immodules-2.0 --update-cache
$ echo "GTK_IM_MODULE=fcitx\nQT_IM_MODULE=fcitx\nXMODIFIERS=@im=fcitx" >> ~/.pam_environment

Make sure you have installed necessary fonts for your language:

$ pacman -S otf-ipafont ttf-hanazono

Now, open up fcitx-configtool and under "Input Method" tab press the little "+" icon in the bottom left corner. Uncheck the "Only Show Current Language" checkbox and search for "Mozc". Choose it and confirm with the "OK" button. I don't usually like GUI tools, but that one is quite handy & quick. You may explore other options too.

Everything should work properly once you relogin.

The default shortcut for changing between input methods in fcitx is Ctrl+Space. If you're normally typing in English, but want to write something in Japanese, you would just press Ctrl+Space, write whatever you want with mozc in Japanese, and then press Ctrl+Space again to get back to your default keyboard setup.

簡単ですね

EBNF

2019-11-20T00:00:00+00:00

BNF stands for Backus-Naur Form which is a notation (or you could say: a metasyntax language) designed for description of context-free grammars. Many extensions and augmentations exist like EBNF (Extended BNF), ABNF (Augmented BNF) or TBNF (Translational BNF).

EBNF is the easiest one to read for humans. Another interesting one is a more complex TBNF, I'd say it's an Extended BNF Extended - it lets you define a structure of the Abstract Syntax Tree and the output Intermediate Code. Seems powerful, but personally I didn't delve into it; for now EBNF will make do.

It generally is a useful tool for language and protocol specification description. You can even use *BNF flavours to generate lexers and parsers automatically (see: Yacc).

EBNF is more or less defined like this:

Notation   Description
    =     - definition
    ,     - concatenation
    ;     - termination of an expression
    |     - alternation (OR)
 [ ... ]  - option, element may be present once or not at all
 { ... }  - repetition, element omited or repeated
 ( ... )  - grouping
 " ... "  - literal strings
 (*   *)  - comments
 ? ... ?  - special sequence, describes something outside of the EBNF grammar
    -     - exception, can be used to exclude something from a rule
    *     - literal repetition (assert 4*"A" == "AAAA")

Here's a wikipedia article with some examples: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form#Examples

Another great practical example is the Go Programming Language Specification: https://golang.org/ref/spec

Go's formal specification played a huge part in the programming language's adoption and even before that - it was crucial for the first implementation.

Reading *BNF is quite intuitive. Writing it, however, is a tedious task.

Coreboot

2019-07-03T00:00:00+00:00

Last time I wrote about libreboot updating, this time I'll write about the coreboot.

The difference between coreboot and libreboot is that the open-source coreboot may need some binary blobs to run properly on some devices, while libreboot embraces the GNU free-software philosophy and refuses to run anything proprietary. Needless to say, coreboot supports way more devices than libreboot. See the coreboot.org website for more info.

I was working with a ThinkPad X220, but the overall process is usually the same. You'll need some way to reprogram the SOIC chip with the firmware. On an X220 the firmware resides on a SOIC-8 chip to which you can connect with a SOIC clip like Pamona 5250 or similar. On the other end of the wire you connect a flasher, for example a RaspberryPi (that's what I used). Long story short: disassemble your laptop, find a proper chip on your motherboard, connect it with a clip to the RaspberryPi GPIO, then you can flash. The coreboot wiki will tell you where on your motherboard the chip is located. Don't get confused: "SOIC clip, SOIC flasher, SPI flasher, SPI programmer" are all just fancy words for the same, basically.

The best guide on how to do all this is: https://github.com/bibanon/Coreboot-ThinkPads/wiki/Hardware-Flashing-with-Raspberry-Pi

To check the GPIO pinout of your particular RaspberryPi:

$ pinout

A little dot on a SOIC chip indicates pin 1, refer to the github page I linked above.

For a SOIC-8, that's the connection you can use:

pin	desc	RaspberryPi
1	CS	(24) GPIO08
2	MISO	(21) GPIO09
3	empty
4	GND	(20) GND
5	MOSI	(19) GPIO10
6	CLK	(23) GPIO11
7	empty
8	3.3V	(17) 3.3 V

You may clean the chip with alcohol before you start. It helps with connection and let's you visually check the exact version of your chip.

Make sure you've properly connected your clip.

Use the latest flashrom.

$ git clone https://review.coreboot.org/flashrom.git && cd flashrom && make

You may have to modprobe some spi stuff. (don't quote me on this one)

$ modprobe spi-{bcm2835aux,gpio,nor}

I'm assuming you are using the RPi GPIO as your SPI programmer. To check the connection:

$ ./flashrom -p linux_spi:dev=/dev/spidev0.0 -V

The common misconception is that you shouldn't use AC power adapter when flashing, but the truth is some chips won't power up without it. Check it out if the flashrom wierdly doesn't cooperate. Again, refer to the github page I linked above.

If the connection was fine, you'll now have a list of chips your motherboard is likely using. Confirm that with the print on the chip casing. Dump the rom image a few times (you can use different -c value) and compare SHA2 checksums. If they're matching, you're good to go.

$ ./flashrom -p linux_spi:dev=/dev/spidev0.0 -c [chip] -r test-01.img
$ ./flashrom -p linux_spi:dev=/dev/spidev0.0 -c [chip] -r test-02.img
...
$ sha256sum test-*.img

Note some informations about your chip, particularly the chip size:

$ ./flashrom -p linux_spi:dev=/dev/spidev0.0 -c [chip] -V

Take one of those dumps, it's your ROM backup now. We will extract binary blobs and deactivate Intel ME with a program called me_cleaner. You can copy that backup and do the rest on your main machine. We will come back to the RPi when we have a coreboot rom image done and ready to flash (NOT YET).

We will flash with a following command:

$ ./flashrom -p linux_spi:dev=/dev/spidev0.0 -c [chip] -w [path/to/coreboot.rom]

Once you flash any coreboot image (unless explicitly set otherwise) you will be able to use the internal programmer and apply instructions from my "Libreboot updating" post in the future; for example to change the configuration.

Now, on how to build a coreboot image... Wiki and even the download page will tell you to build from git, but that's just bullshit - it won't compile. At least it wouldn't for me... Instead, get the latest RELEASE from coreboot.org/downloads.html. Download both source and blobs, then merge folders and open up a terminal:

$ make menuconfig
$ make crossgcc-i386

Most important settings are as follows:

General setup:
- Allow use of binary-only repository
Mainboard:
- Vendor: Lenovo
- Model: ThinkPad X220
- ROM chip size: 8 MB (flashrom told you that)
Chipset:
- Ignore vendor programmed fuses that limit max. DRAM frequency
- Add Intel descriptor.bin file
- Add Intel ME/TXE firmware
Generic Drivers:
- PS/2 keyboard init
Payload:
- (set anything you like, but make sure you'll be able to boot when you flash the rom image)

Now we will need me_cleaner, you can install it from AUR:

$ yaourt -S me_cleaner-git

Let's add the blobs and neutralize Managment Engine:

$ me_cleaner -Srtd [path/to/clean_rom.img] -D [descriptor.bin] -M [me.bin] -O [output_rom.img]

Place descriptor.bin and me.bin in your coreboot directory under '3rdparty/blobs/mainboard/[vendor]/[model]/' like so:

$ cp *.bin 3rdparty/blobs/mainboard/lenovo/x220/

This should be sufficient to start a system.

You might still want to add other blobs, though. For instance - Intel Gigabit Engine may be necessary for some boards to use the ethernet.

Build the final image:

$ make

The output is build/coreboot.rom in the coreboot directory.

Copy it over to your raspberry pi and finally flash!

$ ./flashrom -p linux_spi:dev=/dev/spidev0.0 -c [chip] -w [path/to/coreboot.rom]

If the program prints "VERIFIED", you can safely unplug the clip, assemble your machine and boot.

Rebuild coreboot as many times as you like, tweak and personalise to your heart's content. Now, anytime you want to reflash coreboot, use the internal programmer - no need to disassemble your machine anymore!

CoreWar resources

2019-06-28T00:00:00+00:00

CoreWar is a fascinating game. I don't have enough knowledge nor time to write a proper post about it, but I've had this list of resources lying around on my hard-drive for some time. From here you should depart on a journey to discover by yourself what lies beyond the core.

The original articles written by the game author:

Some hubs:

Warriors' source:

Redcode references:

CoreWar tutorials:

Trivia:

Hills:

http://sal.discontinuity.info/

See you on the hill, warrior!

Libreboot

2019-06-13T00:00:00+00:00

A quick libreboot cheat-note, useful when you want to update (even if it's just your background or config).

Use tools from the libreboot package, from their offcial website: libreboot.org

My version is: r20160907

For flashing you should boot with a following kernel parameter:

 iomem=relaxed

Verify flashrom (it'll tell you whether it's working properly or not):

$ flashrom -p internal -V

Dump current rom:

$ flashrom -p internal -r [path/to/file]

Read rom content:

$ cbfstool [romfile] print

Extract file from rom:

$ cbfstool [romfile] extract -n [filename] -f [filename]

Replace a file in rom:

$ cbfstool remove -n [filename]
$ cbfstool add -n [filename] -f [path/to/file] -t raw

Flash updated rom:

$ flash update [path/to/rom]

That's about it. If you don't have libreboot installed yet, see the installation guide and documentation first: https://libreboot.org/docs/

Before you install, make sure your device is on the list of supported hardware: https://libreboot.org/docs/hardware/

If you're not sure why you'd want to run libreboot, here's a quote from the project's website:

"Many people use non-free proprietary boot firmware, even if they use GNU+Linux. Non-free BIOS/UEFI firmware often contains backdoors, can be slow and have severe bugs. Development and support can be abandoned at any time. By contrast, libreboot is fully free software, where anyone can contribute or inspect its code. [...] Libreboot is faster, more secure and more reliable than most non-free firmware. Libreboot provides many advanced features, like encrypted /boot/, GPG signature checking before booting a Linux kernel and more!"

Moving partitions

2019-04-21T00:00:00+00:00

Moving partitions between different disks or devices. For example: moving /home into a bigger hard-drive.

First move the files. You'll need higher privilege to keep the permission settings.

$ cp -av [source] [dest]

Or with rsync (won't copy files you already have copied), generally faster.

$ rsync -aHAXv [source] [dest]

Carefully with trailing slashes, rsync uses BSD convention whilst cp uses GNU convention.

That's it, if the copy (backup) was all you wanted.

Now we need our new partition to be mounted instead of the old one. We'll achieve that by modifying the /etc/fstab file, which tells the system where which partitions should be mounted at boot time. That is, assuming your partition isn't encrypted. Then it's a whole different story and you probably already know best whether you should tinker with /etc/fstab or rather /etc/crypttab. Let's have a look at our fstab first.

/etc/fstab entry consists of device name or UUID, mountpoint, filesystem type, then options, dump value and pass value. Dump value determines how often the partition should be backed-up by the "dump" program, which we don't use, really. Keep that 0. Pass value is "a number indicating the order in which the fsck program will check the devices for errors at boot time; this is 1 for the root file system and either 2 (meaning check after root) or 0 (do not check) for all other devices." More details here: https://en.wikipedia.org/wiki/Fstab

This file is very important so back it up before modifying.

$ cp /etc/fstab /etc/fstab.old

All we have to do now is modify the proper entry. To obtain necessary information:

$ lsblk -o NAME,SIZE,LABEL,UUID,MOUNTPOINT,FSTYPE

Now simply edit the fstab - that's the easiest way. You could leverage genfstab utility, but that would require additional effort. It's a good practice to leave a comment with partition name and label just above the entry.

Example:

# /dev/sda2 LABEL=home
UUID=2d32c55e-942f-4b45-8694-07029f2583f6	/home		btrfs		defaults	0 2

That's it, you're done. Now reboot and if everything is working - you can delete the /etc/fstab.old backup file.

TTY ricing

2019-02-01T00:00:00+00:00

This is just a post I saved from /g/, thread 66847569; meaning this is not my original content. I thought it was interesting enough to repost here, since there wasn't many other instructions on how to do this. Thanks, Anonymous!

Anonymous Mon 23 Jul 2018 16:40:02 No.66851929:

>>66851873

--- tty ricing guide ---

- install fbsplash or splashutils depending on the distro
- you're gonna need to configure and recompile the kernel
enable required kernel options, see https://wiki.gentoo.org/wiki/Fbsplash
ignore the rest of the article
- install some themes you can use as templates
splash-themes-gentoo or bootsplash-themes or fbsplash-theme-natural-arch or whatever
you can try out some themes like so
splash_manager -c set --theme=THEME
that will set the theme on your current tty, to set the theme on another tty, use
splash_manager -c set --theme=THEME --tty=N
you can't apply themes that don't match your resolution
- make your own theme, it's ez
use a bg img that's in your res, put it in themedir/images, name it "verbose-<your res>.png"
edit the file named "<your res>.cfg" in themedir appropriately
just use another theme as a template, you don't need any guide or whatever to do this
if something isn't obvious, look at <slash>usr/share/doc/splashutils-1.5.4.4-r7/theme_format
- make a simple init script/systemd unit to set your login theme and tty colors on startup
(you can set a different theme in .bash_profile for post login if you want)
- setting the console (tty) font differs between distros, see your distro wiki or whatever
- don't forget to put some cool ascii in <slash>etc/issue

here's how I did it
- <slash>etc/tty_theme - file where I keep the names of the themes I'm using:
export TTY_LOGIN_THEME=login_theme
export TTY_THEME=post_login_theme

- my openrc script where I set the login theme and colors: http://ix.io/1dCa
- relevant stuff from .bash_profile (setting the post login theme):
. <slash>etc/tty_theme
[ "$TERM" = "linux" ] &amp;&amp; [ -z $SSH_TTY ] &amp;&amp; \
 sudo splash_manager -c set --theme=$TTY_THEME

- .bash_logout so when you log out it resets the login theme:
[ "$TERM" = "linux" ] &amp;&amp; [ -z $SSH_TTY ] &amp;&amp; \
 sudo splash_manager -c set --theme="$TTY_LOGIN_THEME"

Anonymous Mon 23 Jul 2018 16:41:22 No.66851939:

>>66851929
had to put <slash> instead of / because 4chan/cloudflare won't let me post if there's unix file paths
use this for tty scrots https://github.com/maandree/scrotty

SQL

2018-12-24T00:00:00+00:00

Typically login with:

$ mysql -p -u [username] [databasename]

With the -p flag you'll be prompted to input password; databasename is optional.

To show databases available for you:

> show databases;

Then you can use the database:

> use [databasename]

Similarly you can show tables in a database:

> show tables;

And describe it's fields:

> desc [tablename];

To find out how to use mysql I highly suggest reading through MariaDB Documentation and Tutorials:

Importing data to the database is fairly simple with well formatted input, but you need to have the database first:

> CREATE DATABASE [databasename];

Database dumps consist of executable commands, so you simply redirect the dumpfile as input:

$ mysql -p -u [username] [database_name] < [dumpfile]

If you have some .txt files (like on matura) you have to create tables yourself accordingly, and then you can load the data:

> CREATE TABLE [tablename] ([columnname] [options_and_datatype], [...]);
> LOAD DATA LOCAL INFILE '[filename.txt]' INTO TABLE [tablename] COLUMNS TERMINATED BY '\t' LINES TERMINATED BY '\r\n' IGNORE 1 LINES ([columnname], @[variable], [...]) SET [columnname] = [expressions];

Mind that in the second statement after the tablename you can skip the rest.

Realistically you'd want to create some users and grant them privileges to work on certain databases. Here's how to do it:

> CREATE USER '[username]'@'[host]' IDENTIFIED BY '[password]';
> GRANT [privilege] ON [databasename].[tablename] TO '[username]'@'[host]';

Databasename and/or tablename can be a '*' wildcard (for all).

You can delete users just like you delete databases:

> DROP USER '[username]'@'[host]';

To show user's privileges:

> SHOW GRANTS FOR '[username]'@'[host]';

Users are saved in 'mysql' database - table 'user'. So to list all users:

> SELECT * FROM mysql.user;

Talking about privileges, I could explain the basic SQL syntax, but I won't. Available privileges: SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, DROP, INDEX, ALL, GRANT OPTION. Privileges for basic queries let you execute those queries. To learn more about privileges specifically: https://mariadb.com/kb/en/library/grant/

SELECT is probably the most important query. Basic syntax:

> SELECT [columnname] FROM [databasename];

Complex query:

> SELECT [columna] AS a, [columnb], [FUNC]([columnc]) AS c FROM [databasename] JOIN [otherdatabase] USING ([commoncolumnname]) WHERE [expr] GROUP BY [columnd] ORDER BY a DESC LIMIT 1;

It's worth mentioning that you can nest SELECT queries, just wrap them in parenthesis.

At that point you might want to learn SQL syntax and database manipulation.

Another important feature of SQL are built-in functions which let you execute conditional statements, manipulate data inline, etc. Example functions: SUM, COUNT, MAX, IF, LEFT, SUBSTR.

MariaDB is capable of encrypting and decrypting AES and DES, it can compress and uncompress, and it can hash stuff with MD5, SHA1 and SHA2. For example:

> SELECT SHA2("[VAL]", [HASH_SIZE]);

This is all I've wanted to write down regarding SQL. I highly suggest you read the docs, both MariaDB and PostgreSQL are wonderfully documented (and they're both great software). For something less specific - w3schools has a tutorial on SQL: https://www.w3schools.com/sql/default.asp

ATmega328P

2018-12-20T00:00:00+00:00

ATmega328 is a popular microcontroller from the AVR family, it's cheap and useful - just the way we like it. It is commonly found in arduinos. Having one you can make an arduino yourself. The foundation of an arduino is it's bootloader. So... the first step to make your very own arduino is to properly plug the atmega chip and flash it with an arduino bootloader. And that's exactly what I did just a moment ago! Let me explain HOW.

As always in electronics, look up the datasheet. There you can find your chip's pinout. Plug GND to GNDs and 5V in into VCC (and AVCC, and AREF) normally and to RESET through a 10k Ohm resistor. If you have a crystal plug it between the XTAL pins and then to GND through 22 pF caps. More datails here: https://www.arduino.cc/en/Main/Standalone

To program the chip you will need 6 pins. We already have VCC, GND and RESET. 3 more to go: SCK, MOSI and MISO. I'm still a newbie in electronics, so I can't really explain to you what all this means and does, but basically we need those pins plugged into corresponding pins on our programmer. The easiest way is to use a different arduino, but you could also try it with RaspberryPi GPIO (~~comming up next!~~ there are multiple tutorials online) or buy a dedicated programmer. I settled for the easiest way, nearly every arduino board has 6 male ICSP pins just perfect for our purpose. And there's an example sketch available in the arduino IDE - ArduinoISP. It's well documented, I suggest you read through all the comments in that code. It could have saved me a lot of time.

from the ArduinoISP sketch

// Pin 10 is used to reset the target microcontroller.
//
// By default, the hardware SPI pins MISO, MOSI and SCK are used to communicate
// with the target. On all Arduinos, these pins can be found
// on the ICSP/SPI header:
//
//               MISO °. . 5V (!) Avoid this pin on Due, Zero...
//               SCK   . . MOSI
//                     . . GND

Just connect those with corresponding pins on your AVR and you're ready to go.

Theoretically it's enough to compile and upload that sketch onto your working arduino, then select proper target (for example "Arduino Nano" board with "ATmega328P" processor) and then use the "Burn Bootloader" option from the IDE. Like in here: https://www.arduino.cc/en/Tutorial/ArduinoISP

But that did not work for me... I switched on verbose output in arduino IDE's preferences and after quite some time of fruitless work I asked for help on IRC (#avrs @freenode).

<wykwit> I'm always getting the Device signature = 0x000000 with atmega328p
<wykwit> what could I be doing wrong?
...
<polprog> make sure the connections are ok
...
<Emil> wykwit: you need 6 pins for spi programming
<Emil> vcc, gnd, rst, sck, mosi, miso
<Emil> Depending on the burned fuses, you might need a crystal of appropriate frequency on the xtal pins
<Emil> you also might need to use the low speed programming if first if you have the div/8 fuse burned (by default it is)
<wykwit> lowering the SPI_CLOCK in ArduinoISP script worked just fine
<wykwit> thanks
...
<polprog> wykwit: yeah thats a good trick. works 9 outta ten times ;)

You should REALLY CAREFULLY read the comments in the sketch and adjust values accordingly, so everything works fine for you. Pay attention especially to SPI_CLOCK and BAUDRATE settings.

I suppose the first flash could work with the default SPI_CLOCK and BAUDRATE of 1000000 (3rd option). Try it out. I finally flashed with SPI_CLOCK set to (1000000/32) and a default BAUDRATE of 19200.

You could probably get those magic values from the datasheet, but the datasheet is more than 600 pages long, so I just dug some values up from the internets and experimented.

For those who are curious about The Real toolchain and how all of this works under the hood I highly suggest reading documentation at: http://www.nongnu.org/avr-libc/

Now I'll have to look into fuses. I hear they are important, but we automatically set them when flashing arduino bootloader.

If you want to keep hacking your ATmega328 with the arduino IDE this chart might be helpfull: https://www.arduino.cc/en/Hacking/PinMapping168

Now you can simply keep uploading new sketches through your current arduino setup with the "Upload Using Programmer" option.

AUR

2018-04-30T00:00:00+00:00

AUR - Arch User Repository

Something every Arch Linux user MUST be familiar with. In this post, I'll describe how to install yaourt, a pacman wrapper with AUR support, from the AUR. That way you'll know how to work with AUR and you'll have a tool which will relieve you from ever doing that AUR stuff manually again.

https://aur.archlinux.org/packages/yaourt/

Update: yaourt is not the coolest kid in town anymore. Instead check out yay or paru.

First of all: dependencies. Dependencies are packages needed for a certain package to work. We need to resolve them manually, installing them before yaourt. They're always listed on the packages page. To install packages, marking them as dependencies we'll use:

$ pacman -S --asdeps [packagename]

You may have some of package's dependencies already installed on your system. To skip reinstallation process use '--needed' flag. So for our yaourt we would issue a following command:

# pacman -S --needed --asdeps diffutils gettext pacman rsync

Note that one of yaourt's dependencies, namely package-query, is not in the official repositories and that's why I didn't include it in the command above. It is (also) available in the AUR! We have to install package-query before we can proceed. Fortunately all it's dependecies are in the repos. Now we'll install our first package from AUR - package-query!

https://aur.archlinux.org/packages/package-query/

For that we'll need a PKGBUILD. It is a file describing build and installation process of which proper programs then take care. Historically AUR was just a database for those PKGBUILDs. Now it's a little bit more complex; links we'd need are in the upper right corner of the AUR package page.

$ curl https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=package-query -o PKGBUILD
$ makepkg -si

makepkg will make a package, -s flag will (once again) check for dependencies and resolve them with pacman, and -i flag will install the package (with pacman) when it's ready. As you issue these commands the package should be installed.

Now you can repeat the process for yaourt, safely omitting -s flag; we've already taken care of dependencies.

$ curl https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=yaourt -o PKGBUILD
$ makepkg -i

If everything went right you'll have working yaourt installed now. From the user perspective it works just like pacman, but includes AUR. Now you can install AUR packages with just a:

$ yaourt -S [packagename]

Search through AUR and repositories with:

$ yaourt -Ss [search term]

And update everything, including AUR packages with just a:

$ yaourt -Syua

Now, when you query for orphans (unused packages):

$ yaourt -Qdt

yaourt will ask you if you want to remove them, which of course you do. Pretty convenient.

You can work with yaourt like you would with pacman (it's just a wrapper). For instance, to remove something:

$ yaourt -Rncsuv [packagename]

This will give exactly the same result as:

$ pacman -Rncsuv [packagename]

Hope you like your new toys. Have fun!

GPG keys

2018-03-21T00:00:00+00:00

So you've encountered "unknown public key" error message. What to do?

First search for the unknown key. Additionally you can explicitly specify the keyserver, for example keyserver.ubuntu.com

$ gpg --keyserver keyserver.ubuntu.com --search XXXXXXXXXXXXXXXX

If you see a key that is reliable just type in its number and it'll be imported. Error should be gone.

GPG keys are usually used to sign and/or encrypt e-mails. To generate you own keys type in the following commmand in your terminal and simply go through the generation process.

$ gpg --full-generate-key

Now you'd like to submit your key on a keyserver or share it yourself. You'll need to paste generated files content onto your keyservers website.

$ gpg --export --armor [your e-mail address] > [filename]

Someone can then import it from the keyserver, like we did above, or from the file:

$ gpg --import --armor < [filename]

You can delete keys with

$ gpg --delete-keys [searchterm]

and your private keys with

$ gpg --delete-secret-keys [searchterm]

Where the [searchterm] is either e-mail address, key ID, or really anything that identifies your key.

You can list keys with:

$ gpg --list-keys

Great, but if you now receive an e-mail signed with a valid signature (which you've imported) you'll get a warning "cannot verify sender". To fix this, change the level of trust and sign the foreign key.

$ gpg --edit-key [searchterm]
gpg> trust //Here I suggest 3 or 4. Use 5 only for your own keys.
gpg> sign
gpg> save

It'll work now.

You can move your private keys across different computers. Don't lose your private keys or else you won't be able to decrypt your e-mails. If needed (lost, stolen, stale), keys can be revoked.

SSH keys

2018-03-20T00:00:00+00:00

To generate a key-pair issue:

$ ssh-keygen

This will produce two (2) files of a name provided on the prompt: [name] and [name].pub

[name].pub is what goes on the server, more on that below. File [name] is your secret key, which you'll use instead of password for authentication. Transfer files between machines either with scp or sftp, whichever you like.

To use a key on ssh:

$ ssh -i path/to/your/key [username]@[host]

You will be asked for the password. Not the user password, but the (optional, though highly recommended) password which you've inputed during the key-creation.

"[username]@" part can be skipped when $(whoami) == [username]

In case you don't know yet - Putty keys ain't compatible with OpenSSH, but you can convert those keys both ways.

As for the server settings (/etc/ssh/sshd_config), you need:

~ PubkeyAuthentication yes
~ AuthorizedKeysFile .ssh/authorized_keys

And I suggest:

~ PasswordAuthentication no

Defaults for the rest are fine.

Now you need to place your [name].pub (public key) file on the server under: .ssh/authorized_keys

That's the most common mistake - IT IS A FILE and the filename must match with your sshd_config.

$ cat [name].pub >> ~/.ssh/authorized_keys

This will only append to the file. Adding a new key.

$ mv [name].pub ~/.ssh/authorized_keys

This will overwrite what you've had saved in the file, making old keys unusable (which is good).

Multiple keys can be used for one account and one key can be used for multiple accounts. Keys are stored in a separate directory for each user. You could use direct path to specify one authorized_keys file for the whole system. For more information RTFM & STFW.

Static IP

2018-03-13T00:00:00+00:00

Network configuration - static IP. Useful for installations and repairs, when dhcp does not work as intended.

https://wiki.archlinux.org/index.php/Network_configuration#Static_IP_address

I use it way too much.

Display all the info with a following command:

$ ip a

Bring the interface up:

$ ip link set eth0 up

Set the IP:

$ ip addr add 192.168.1.13 broadcast 192.168.1.255 dev eth0
$ ip route add 192.168.1.1 dev eth0
$ ip route add default via 192.168.1.1 dev eth0

You may try that, instead of these two last commands above:

$ ip route add default via 192.168.1.1

Delete all address settings for a given interface:

$ ip addr flush dev eth0

A newer way to set the IP:

$ ip address add 192.168.1.13/24 broadcast + dev eth0

Of course replace 'eth0' with whatever your interface is called (for example: enp1s0) and properly adjust all the IPs.

cstdio

2018-03-06T00:00:00+00:00

Using printf() & scanf() in C/C++ code is not only faster than using cin & cout, but it's also way simpler and cleaner.

To use scanf/printf include 'stdio.h':

#include <stdio.h>

or 'cstdio' in C++:

#include <cstdio>

For cin & cout it's 'iostream':

#include <iostream>

cin inputs whatever the fuck you want into a variable. There's no difference in syntax, whether your input is a string or a single integer. That means at the first glance you won't know what to expect from the input. Syntax:

cin >> [variable] >> [another variable] >> [etc.];

That way you can get the input into 3 (or more) variables. For instance, this might input 3 different numbers into 3 different vars, but might as well get you a number, a string, and a char.

cout works similarly, it spits out your variables to the output. To input more variables (of different types?), you need to add even more "<<". AFAIK that is necesarry. So, if you want to output a few numbers, each separated with a few characters, you need to add "<<" in between every single part of your call. Syntax:

cout << [vars] << "or text\n " << [more vars] << [etc.];

Notice, that the first variable won't be spaced out from the string that comes after it. On the output you might get for example "1321or text"...

To clear out: '\n' means newline. Escaped characters work both with couts and printfs. Some others are '\t' for tab and '\a' for a beep sound.

Now to the printf(), as it's simpler to output, than to input.

printf() takes a string(char table in C) and parses it, replacing special character sequences with variables, which you pass to the function as later arguments. This might seem confusing for you C++ monkeys, but it's really not. Syntax looks like this:

printf("yo %[type code], simple, ain't it? %[same] %[etc.]\n", [var1], [var2], [etc.]);

The string has to be there, or else you'll get a compilation error. By the [type code] I mean one of these sweeties below:

%d - int (same as %i)
%ld - long int (%li)
%lld - long long
%f - float
%lf - double
%c - char
%s - string
%x - hexadecimal

As you've probably already figured out, you need to supply which type you want to output on which place. This %something is a placeholder for the value of our variable. Such approach has a number of benefits. First of all, you know what type you're outputting/inputing on the first glance. Secondably - it's much simpler to space out variables with whatever you like. Mixing and formatting output is just way easier with printf.

scanf() works similarly, but there's a small difference. You shouldn't supply your variables straight to the function, that won't tell it anything. You pass the space where you want your variable to be saved, so an address of a variable, a POINTER. Simple as that. If you don't know ANYTHING about pointers and memory addresses, just remember that with every type, except for tables (particularly char tables/strings), you need to preceed your variable name with a single '&' character. This will pass the address instead of the value further to the function. The syntax is as folows:

scanf(" %s %[?] %[?]", [stringname], &[var1], &[var2]);

passed string typically consists just of spaceholders and spaces. By a space, scanf means every white character. In this case, it'll ignore every '\n' and ' ' preceeding our string.

To summarize:

// input with a format string (to a string)
scanf("%s", stringname);

// input with a format string (for other types)
scanf("%?", &varname);

// output with a format string
printf("%?", varname);

That's basically it. An example:

#include <stdio.h>

int main(void) {
	int n;
	scanf("%d", &n);

	int tab[n];
	for(int i = 0; i < n; i++)
		scanf("%d", &tab[i]);
	
	for(int i = 0; i < n-1; i++) {
		printf("%d, ", tab[i]);
		if(i%5 == 4)
			printf("\n");
	}
	printf("%d.\n\tAnd that's it.\n", tab[n-1]);

	return 0;
}

BONUS: to get higher i/o speed with cin/cout use following lines in your c++ main function:

std::ios_base::sync_with_stdio(0);
std::cin.tie(0);
std::cout.tie(0);

RTFM

2018-01-21T00:00:00+00:00

RTFM = Read The Fucking Manual

How to know which parameters to use with a command?

In GNU/Linux you can usually issue a command with '--help' parameter, which will display a help page for you. If not, try '-h', '-H', '-?' or none parameters.

You can read man(ual) pages with:

$ man <command or keyword>

If you're not sure what you're looking for, try:

$ apropos <search term>

If you don't know what a command does, there is:

$ whatis <command>

and on Arch Linux (for packages):

$ pacman -Qi <packagename>

on Debian-based distros that would be:

$ apt show <packagename>

For online man pages check 'info' command, it seems to work just like 'man'.

If you don't know where a certain program is, you can use:

$ whereis <name>

If you're still clueless STFW.

STFW = Search the Friendly Web

Try to avoid proprietary search engines like Google or Bing if you can.

I suggest using: https://searx.me/

And https://stackoverflow.com/ is your friend.

Here are some more references: http://www.catb.org/~esr/faqs/smart-questions.html

Read before you post.

Underclocking

2018-01-20T00:00:00+00:00

Program you're gonna need: cpupower. It's in the official repository.

More info: CPU frequency scaling on the Arch wiki.

You can probably figure out the rest on your own...

Have a look at: /etc/default/cpupower

$ cpupower frequency-info
$ cpupower frequency-set -g powersave
$ cpupower frequency-set -u 3.1GHz
$ cpupower frequency-set -u 2.8GHz