A Brief History of Brazil’s Electronic Voting System (for the nerds in the room)

While 2020 has been a challenging year, working remotely with an improvised home office has had its perks. For instance, I found the time to write this article in English hoping it can serve as a brief introduction to the idea that electronic voting ain’t all that bad, and how it has been successfully used in Brazil.

In a brief history about me, I’ll refrain from too many details. Suffice to say that I majored in Computer Science 20 years ago. I have worked as a programmer for various companies and multinationals, and now occupy a position of Systems Administrator for a local state government institution in Brazil, handling several aspects of application infrastructure, environment maintenance and provisioning. Far from being a security expert myself, I can confidently understand the technology involved, technical terms, the theory and tools behind it.

E-Voting is a bad idea… or is it?

More often than not, discussing the idea of electronic voting, someone is bound to throw you links to Tom Scott’s videos, which, to be fair, are excellent and valid arguments against it.

But I am here to present somewhat of a challenge against those arguments. Not just mere conceptual arguments, but concrete facts about how Brazil managed to get a firm grip on its electoral process, going from a collapsing mess of paper ballots and rampant corruption in the 1990s to, arguably, one of the fastest and most reliable voting systems in the world. That was only possible with constant government investment on a process that relies, not only in a custom designed hardware and software, but also a combination of transparent protocols and manual safety measures that almost completely thwart the uncertainties brought up by Scott in his videos.

One Institution To Rule Them All

The Brazilian Supreme Electoral Court (TSE) was created in the 1930s to regulate and control all aspects of the electoral process. It is responsible for creating and enforcing regulation on how the country, states and cities should conduct elections, from electoral campaings, funding, voter registration, district boundaries, all of it. One set of laws to rule them all.

Governance at the federal level is even more difficult when combined with the fact that voting in Brazil is mandatory by its Constitution. It is not just a civil right, it is a civil duty. Refusing to vote without justifying your absence is punished with restrictions on other civil and financial aspects of your life. Keeping tabs on aproximately 147 million voters considered to be automatically enrolled as soon as they turn 18 is quite a challenge. It is not difficult to understand why the US never bothered to even try.

Thwarting Election Fraud

By the early 1980s election scandals and voter fraud were rampant and spreading like wildfire. It became quite clear that something needed to be done at the imminent collapse of the TSE, at great cost to Brazilian democracy. But instead of throwing in the towel and giving up control to more independent state level governments, the TSE decided for a bet on innovation. It aimed its own IT department into a taskforce, along with the Aerospace Technology General-Command (CTA) and the National Institute of Space Research (INPE). A group of highly specialized professionals worked together to conceptualize an electronic voting system with security as its absolute most valued requirement. The first election to partially use the new electronic machine was held in 1996, and by the year 2000 every paper ballot was effectively replaced by the electronic vote.

The Electronic Voting Machine (CEV)

Over the years, several iterations and improvements were made to the original project, but overall this is what is looks like today.

TSE investe em 180 mil urnas eletrônicas para as eleições ...
The Brazilian electronic voting machine. It’s not a fancy tablet. It’s not a laptop. It’s not a gaming rig.
Every state and city uses the exact same model, purchased and maintained with federal funds.

The machine itself is not a random piece of equipment bought from any manufacturer. A custom hardware design was implemented to be strictly followed by government contractors. The exterior design was especially crafted to make it as simple as possible to use, and directly incompatible with the usual market standards, such as HDMI, USB or any off-the-shelf card readers. Everything fits inside a hardened metal case, which is physically sealed with TSE markings to prevent tampering. This includes the monitor, keypad, a small printer, emergency battery, power brick, etc.

After a public audit, the CEV is physically sealed to make sure it
cannot be tampered with during the election process.

The only hardware that sticks out from the metal case are the power cord and another limited keypad used by selected technicians to control, reset and prepare equipment as voters are allowed into a protected booth to cast their vote.

Dia D: eleitorado de Manaus têm 11 opções a prefeito e 1 ...

A few things to note right off the bat:

  • The entire case is designed to be metal and difficult to penetrate using sharp objects. Internally, metal plates below the screen and keypad also protect internal components from hard falls or people trying to break in with an external force. It is designed to be sterdy and difficult to open without any proper tools.
  • The external keypad used by technicians has limited funcionality and purpose.
  • All cords are extra resistant and physically embedded so they cannot be unplugged without damaging the equipment.
  • Modern iterations of the external keypad now have a fingerprint reader so voters can be identified with a 2nd factor before allowed into the voting booth. Thanks to a nation-wide regulatory agency, like the TSE, all voter records are kept in one database and citizens are required to keep things up-to-date every 2 or 4 years. Other government agencies now feel the urge to tap into what is considered the largest, most up-to-date civil database in Brazilian history, apart from the Brazilian IRS.
  • No network interfaces. No ehternet. No Wi-Fi. None of it. The CEV is completely cut off from any and all networks. The only access to its memory is done via a custom USB port and its main memory card, which are physically sealed in during the election.
  • No fancy touch screens. The robust and intuitive physical keypad was chosen because that’s what most people are familiar with. It looks like a phone keypad.
  • No complicated user interfaces. During the campaign, all candidates are assigned a unique number that identifies them. Voters are only required to input the candidate’s identification number and confirm. They are encouraged to write those down and bring a cheet-sheet on election day to remember. Press GREEN to confirm, RED to correct, WHITE to vote for nobody.
Fausto Peres 15.630 | Com a Política a gente não brinca!
An example of what the voter sees after entering the candidate’s number and pressing GREEN. A profile is displayed and the voter confirms by pressing GREEN again. The voter repeats this for every seat in dispute (such as president, governor, mayor) and that’s it.

The Software and Operating System

In its current version, the CEV runs a highly customized, stripped version of Linux, compiled specifically for this purpose. There is absolutely no proprietary software installed, and all components are recompiled from opensource. Once every unecessary driver and package is removed, all components and files that remain are hashed and digitally signed with the TSE’s in-house Certificate Authority and the keys from political parties as well. The digital signatures play an important role into the general election process, as we’ll see further down.

This Linux distribution is managed by the TSE IT Department itself, inluding the in-house development of the counting software, which tabulates and counts the votes while trying its best to keep the integrity of voter privacy and vote secrecy required by the Brazilian Constitution.

The Election Process In a Nutshell

Audit Ceremonies

Before the election, representatives from all political parties, the Brazilian National Lawyer Association (OAB), the Public Prosecutor’s Office (MP), a number of Universities and several other agencies are all invited to participate in a number of pre-election audits. This includes policital representatives, as well as software technicians with enough skills to validate and confirm what the TSE is actually doing.

180 days before election, an audit is held at TSE headquarters where all involved parties are allowed to inspect and review the source code and documentation. The moment can be used to suggest any changes or improvements. Technicians are allowed to bring their own equipment and software to run any validation they want in the source code.

20 days before election, another ceremony is held where the same source code and components are again opened for inspection. This time, the source code is compiled on premise to produce the artifacts that will be installed in every voting machine.

Once compilation is done, a TSE technician runs another software routine to create a hash for every artifact that will be included in the CEV. At this point, public encryption keys from all political parties and major state agencies are gathered. The encryption keys are used to collectively sign all hashes to make sure nothing can be altered without everyone’s keys being involved.

While one may have doubts about the software running in the TSE technician’s workstation, every other technician in the room is allowed to bring their own physical equipment, use their own hashing and encryption software to rerun and validate and double check what is being done at the TSE’s technician workstation.

Once everyone is satisfied with the results, the signed artifacts are flashed into a few portable storage media cards. One copy will be kept in TSE’s safe and one master media will be used to flash every voting machine just before the election.

Every hash and digital signature created in this ceremony is published on the TSE’s website and available to the public. They can be downloaded and used in other audits that occur during and after election day.

Roughly a week before election day, all machines are flashed with the signed software, sealed from all sides, and sealed into boxes to prevent any tampering. All of 500 thousand CEV units are now ready to be shipped to voting locations across the country.

Election Day

Once voting machines are distributed, boxes are only opened at election polls, early in the morning of election day. For every opened box, another ceremony is held with local officials and representatives of political parties. The CEV is booted for the first time after being flashed. Using only the attached external keypad, the CEV is reset and all counts are cleared. The embedded printer spits out what is known as the Zero Report (Zerésima) showing the total count for all candidates as zero.

As noted before, voter registration and voter ID is somewhat automatic once a Brazilian turns 18. Citizens are required to show up at a local TSE facility (known as TREs) to update the record of where they live and scan their fingerprint for biometric 2FA. From that, the TSE system will assign you to a voting poll closest to where you live.

Worthy of note, here, is the fact that voting locations are not at all subject to the whims of elected politicians. Brazilian law firmly states that you vote exactly where TSE assigns you to vote, and that is always the poll that is closest to your residence. You simply do not get to choose. More importantly, politicians never get to chose either. This completely eliminates gerrymandering and the manipulation of district division. Every vote counts, and every vote counts equally.

Because the TSE controls and knows before hand where everyone should cast their vote, each voting machine is pre-loaded with the exact list of voters that are allowed in that district. When you show up at the poll, a group of technicians will be there to prepare the voting machine for you.

Mesários começam a ser convocados para eleições - Tribuna de Ituverava

Their job is to double check your identity and input your voter ID into the external keypad to activate the voting machine.

It might seem like a privacy violation is happening before your eyes, since a voter ID could be easily matched against the vote inside the machine. However, several audits have verified that the voting software is clever. It scrambles the internal counting table and does not allow any count to be traced back its voter ID. The total count is all that matters.

At the end of the day, local representatives are called back for a closing ceremony. The CEV is instructed to print the results of its count. This paper copy is not used for the official count. All representatives get their own copy and another is physically posted at a public board for everyone to check local results.

Notebly, the printed paper copy does not only contain local results, but also a digitally signed QR code with the same information. Anyone with a smartphone can scan the QR code, and later compare the results that are eventually posted at the TSE’s website. A number of civil initiatives attempt to keep tabs on the election results that way, urging everyone to snap a photo of their poll’s QR code and upload it for an independent count.

The actual digital contents are obtained during the closing ceremony, where one seal is broken to extract the memory card where results are stored. The contents of the card are also digitally signed. And, because the voting machine itself is not connected to any network, the contents are uploaded using another computer at hand, which is also not connected to the Internet. A private secure network is used exclusively to upload the results to services hosted at the TSE’s datacenter. If any connection problems are found, the memory card or the CEV unit itself can be physically brought to a local TSE facility, where the contents can be transmitted.

Audits, audits, audits

Throughout the entire election process, any institution or political parties involved are invited to question and contest the results, if they feel that is necessary.

Even after the election is closed and decided, a public audit can and is usually conducted. For those, a number of random CEV units are selected from storage. The equipment is physically opened and dismantled so that everyone involved can inspect it, including its internal parts. The software can be verified and all signatures can be matched against the keys gathered during the Audit Ceremonies before the election. The contents of the count made by the unit can be matched against the results published on the Internet.

In the recent 2014 election, a highly controversial and unexpected result shocked a divided country. The opposing party that lost immediately requested a complete audit and recount of the results. While the recount was quickly achived among various software validations, over the next few months not a single evidence of fraud or tampering was found.

Public Security Audits

Every election year, before the election, the TSE also conducts public audits of the CEV design, where IT specialists are invited to blackbox testing and report any vulnerabilities they find. As a result, a number of security flaws have been found and fixed over the years.

Parallel Voting Simulation

On the eve of election day, a number of CEV units, loaded with the official election software, are randomly selected. They are used in a simulated parallel voting session, where auditors openly vote without any secrecy so that results can be manually verified at the end. This parallel audit is conducted in all major cities and a few others in every state.

Is it perfect, thought?

Of course, it’s not. But the argument that electronic voting is a bad idea all around and should never be considered, IMHO, sounds more like a paranoid thought experiment.

Conducting secure electronic voting will never be easier or cheaper than paper ballots and manual counting. But, is it worth to disregard the benefits? If you are asking yourself how much does it cost, then you should also be asking what is at stake? Election fraud in Brazil has dropped to an all time low, and the full results of a nationwide presidential run can be broadcast later on the same day, just in time for the nightly news.

There is a lot to improve in the process we use today. I think the secret to the sucess of Brazilian elections, is the fact that the process does not rely entirely on electronic machines. The TSE knows there are paths to corrupt an electronic election, and they work hard to mitigate them by implementing manual protocols and safety measures, using digital and analog tools in a smart way. The voting machine itself is electronic and it counts really fast, but everythine else around it is still analog and strictly controlled by a large number of real people.

If you ask any Brazilian today if they would give it up, I doubt you will actually find anyone that wants to go back to counting paper ballots for weeks in a row in city sports gyms around the country.

Computadores e a Internet não são mágicos… e nem é tão difícil assim de explicar

Venho trabalhando com tecnologia e desenvolvimento de software a pelo menos 20 anos; mais recentemente na área de infraestrutura e operações. Ao longo dos anos, como muitos colegas da área, encontrei pessoas que, considerando eu como sendo o “nerd da turma”, questionam: Como é que funciona? Quando dizemos “fazer download“, “compartilhar no WhatsApp” ou “publicar no Instagram“, o que realmente acontece lá dentro do computador?

Muito dessa curiosidade nasce quando a pessoa percebe que, apesar de tantas comodidades, não entende bulhufas de como os aparelhos mais modernos funcionam.

Para a pessoa comum, o computador é literalmente uma caixa mágica com botões e uma tela colorida. É uma máquina surpreendente com a qual podemos jogar, compartilhar momentos, comprar coisas, conversar com amigos, encontrar pessoas… Mas assim que um “especialista” tenta explicar como ela funciona por baixo dos panos, a conversa desanda muito rápido. Tentativas variam desde explicações sobre o sistema binário até as mais elaboradas analogias, como tubos e encamento. A maioria logo descamba sobre rebinbocas e parafusetas que, para um leigo, não tem pé nem cabeça.

Certamente, é um dos motivos de, hoje em dia, tantos filmes mostrarem pessoas destruindo o computador quebrando a tela do monitor.

A analogia que prefiro utilizar é relativamente simples. Tenta explicar como computadores e a Internet funcionam de tal forma que uma pessoa leiga possa entender, ao mesmo tempo que tenta manter os conceitos fundamentais do que acontece dentro da máquina. Não chega nem perto de falar sobre operações matemáticas, memória principal, dispositivos de armazenamento ou interfaces de rede. Esqueça tudo isso. Mantenha o foco no propósito das coisas.

Quando você compra um computador, na verdade está comprando um baralho

Quando você vai até uma loja e compra um smartphone ou notebook para trabalhar, sem sombra de dúvida, gastou dinheiro para ter o que pode ser considerado o ápice da evolução tecnológica da humanidade; um aparelho delicado e complexo que manipula a eletricidade para realizar bilhões de operações a cada segundo.

Por baixo daquele estilo e aparência modernos, os milhares de pequenos componentes eletrônicos lá dentro escondem um conceito muito mais simples do que se imagina; uma idéia que passa despercebida para a maioria das pessoas.

Jeu de cartes sur ardoise... bannière

Agora, em casa, com seu novíssimo baralho, o que pretende fazer com ele? Obviamente, pode jogar qualquer jogo de cartas, mas você é livre para fazer o que quiser com ele. Cartas são objetos físicos, são sua propriedade e pode usá-los como quiser. Pode pintar sobre sobre elas com canetinha, montar um castelo, contar quantas cartas o baralho tem, pode agrupar as cartas em cima da mesa para fazer uma belo desenho. Você escolhe.

Imagine que tenha colocado as cartas em cima da mesa, formando um belo padrão de cores. É uma obra de arte!

Deck of playing cards on blue background. Gambling concept

Imagine, também, que queira compartilhar essa obra com seu melhor amigo. O problema é que ele mora a centenas de quilômetros de distância, mas você quer que ele veja isso o quanto antes. Uma opção é pegar o telefone, ligar para ele e passar instruções para que ele possa remontar o desenho com algum outro baralho que ele tenha. Quando terminar de passar as instruções, ele terá uma réplica do seu desenho.

Agora… Vamos parar um pouco e pensar sobre alguns pontos importantes.

Seu amigo não tem o seu trabalho original. Nenhuma carta na mesa dele é sua. O que ele tem não é uma foto, não é uma cópia xérox, nem mesmo uma descrição detalhada. Ele realmente tem uma réplica exata. Mais importante ainda, perceba que a sua obra original nunca deixou sua mesa. Você continua com todas as suas cartas, na mesma ordem e na mesma quantidade. De forma literal, nenhum objeto físico foi movido de uma cidade a outra. Nenhuma imagem foi transmitida pelos cabos telefônicos ou por algum tipo de “encanamento mágico“. A única coisa que foi transmitida pelo telefone foi a sua voz, ou seja, instruções de como recriar o seu trabalho. Já que você e seu amigo falam a mesma língua, ele pode entender as instruções e criar uma réplica perfeita a centenas de quilômetros de distância. Nenhum talento necessário!

Ok, pera aí. É só isso?

Sim, realmente é. Confie! Meu diploma em Ciência da Computação pode ajudar a confirmar isso.

Pense na conversa telefônica que você teve com seu amigo como “sua conexão de Internet” e baralhos como sendo “computadores”. Cada pessoa tem seu próprio computador e dentro dele você tem pequenos blocos que podem ser organizados de qualquer forma, como se fossem cartas de um baralho.

Se você já tentou imaginar o significado da expressão “informação digital”, as cartas do baralho são o melhor exemplo. Informação pode ser representada por pequenos blocos que podem ser manipulados e ordenados individualmente. Basicamente, todos os blocos são iguais. Assim como as cartas de um baralho podem estar viradas para cima ou para baixo, “blocos digitais” são pequenas peças eletrônicas que podem estar ligadas ou desligadas. A única coisa que importa para fazer sentido da informação que eles guardam é o propósito e a ordem em que são colocados.

Informação digital nunca chega a lugar algum porque, de fato, ela nunca sai do lugar de origem. A Internet realmente é uma rede de sinais eletrônicos, seja com fio ou sem fio. Tudo que se transmite são apenas instruções de como criar réplicas exatas do conteúdo original, efetivamente “copiando” a informação de um computador para dentro de outro.

Caso queira debater de forma sensata e concreta qualquer assunto relacionado à era digital em que vivemos, esse é o conceito mais básico que precisa ser compreendido antes da discussão. Torna-se relevante quando se percebe que todos os computadores do mundo são construídos dessa forma e todos funcionam exatamente do mesmo jeito.

Digamos que a discussão gira em torno de criar ou atualizar leis de direitos autorais, patentes, neutralidade da rede, privacidade, crimes cibernéticos, crypto-moedas… Seja qual for o assunto, lembre-se do baralho, pois tudo na Internet acontece dentro daquele mesmo cenário.

Sempre que ouvir uma sopa de letrinhas ou expressões complicadas de um especialista tentando explicar como essas coisas funcionam, lembre-se de que são apenas palavras bonitas que eles mesmos inventaram para organizar a complexidade e as dificuldades de construir máquinas que, no fundo, fazem apenas isso: processar e replicar informação de um lugar para outro.

Computers and the Internet are not magical… and it’s really not that hard to explain

I’ve been working in the tech industry, close to software development, for about 20 years; more recently at the infrastructure and operations level. And throughout time, like many of my fellow colleagues, I’ve met a number of people who looked at me as “the computer guy” and decided to ask: How does it actually work? When we say “downloading a video“, “re-tweeting a joke“, or “sharing a picture on Instagram“, what is going on in there? What exactly is coming in and out of those “Internet pipes“? This curiosity usually sparks after they feel shamed by one of Dara O’Brian’s comedy stand ups.

To the common user, computers are literally magic boxes with buttons and a color display. It’s a wonderful machine you can use to play games, share moments, buy stuff, talk to friends, meet new people… But when a tech savvy person tries to explain how anything works under the hood, things go south really fast. Attempts to explain vary from how binary numbers work to all sorts of analogies, down to trucks and a series of tubes. Most of the time, an in-depth explanation quickly devolves into a Rockwell Retro Encabulator presentation.

No wonder so many movies depict people destroying computers by trashing the monitor. People just give up, and social life keeps going.

The analogy I tend to use is somewhat simpler. It tries to explain how computers and the Internet work in a way that’s simple enough, but still captures the fundamental concept of what is going on under the hood. This analogy makes no attempt to dive into complex operations performed inside the CPU, or how bits of raw data move between cache registers, main memory, permanent storage or network interfaces. Forget the Retro Encabulator for a moment and focus on the actual purpose. Remember what it all means. What it’s actually used for.

Buying a computer is like buying a deck of cards

When you go to your favorite store and order a brand new device — a smartphone or a laptop you can work on — no doubt, you are actually buying what can be considered the apex of human technological evolution. A delicate, complex machine that uses electricity to pump billions of operations in one second.

But, behind that shining exterior and the fancy tech jargons you see on the box, those tiny electronic components inside hide a much simpler concept; one that most people take for granted.

Jeu de cartes sur ardoise... bannière

So… now at home, you have your brand new deck of cards. What can you do with it, aside from the obvious choice of playing games? You are free to do whatever you want. The deck is a physical object. It is truly yours. Paint over the cards, build a castle, count the cards, arrange them on the table to form an beautiful pattern. The choice is yours.

Imagine you managed to arrange all of the cards on the table to create some pattern that you like. It’s a work of art!

Deck of playing cards on blue background. Gambling concept

Let’s say that you really, really like what you see and immediately want to share it with your best friend, except he lives hundreds of miles away. The fastest move would be to pick up the phone, call him and pass on instructions so he can recreate the same pattern using his own deck of cards. When you are done passing instructions, your friend will have a replica.

Now… Here are some important things you need to take notice right now. Your friend does not have the original work you created. Not a single card on his table is yours. He does not have a picture of your work, not a fake mockup, not a carbon copy, not even a description. It truly is an exact replica. Most importantly, you should realize that your own creation never left your house. You still have all of your cards with you, lined up the same way as before. No image or any physiscal object was literally “transmitted over the wire” or some “magical tube“. The only thing that went over phone was your voice; instructions on how to recreate your work. Since you and your friend speak the same language, he can understand the instructions and create a perfect replica from miles away — no talent required!

Wait, that’s it?

Yes. It really is. Trust me! My Computer Science degree tells me that. Think of the “phone conversation” you had with your friend as your “Internet connection” and “decks of cards” as “computers”. Each person has its own machine, and is free to rearrange their own contents anyway they want.

If you ever wondered what digital information actually means, that concept is perfectly layed out in your deck of cards. Information we care about is represented using smaller pieces that can be managed individually. Fundamentally, every piece is identical in nature, and the only thing that matters to make sense of them is the order and the purpose for which they are presented.

Digital information never arrives anywhere because it never leaves. The Internet really is a web of electronic signals, wired or wireless. Those signals are instructions to recreate exact replicas of the information from one computer inside another.

If you ever want to have a sensible discussion about anything in the digital era we live in, that basic understanding needs to be clear. It becomes relevant when you consider that every computer on Earth is built the same way, and works the same way. For example, if we want to write better laws for copyright infringement, patents, net neutrality, social privacy, cyber crimes, crypto currency… Remember the deck of cards, because that’s the underlying scenario on top of which all of them actually happen.

Everything else you hear from experts are just fancy words for complex ways they found to manufacture machines that operate on that same simple idea.

There is a way to create valid Let’s Encrypt certificates for local development… sometimes

Caveats and a quick Docker example included 👍

A common developer trick that’s become popular is to work with valid domain names handled by magic DNS resolvers, like nip.io, xip.io, or sslip.io. If you include any IP address as part of their subdomain, their DNS servers will resolve to that IP authoritatively.

For example, you can use the domain 127.0.0.1.sslip.io anywhere and it will resolve globally to 127.0.0.1. The IP address itself is not available anywhere except your local machine, but the domain is considered a valid, resolvable name worldwide. This gives developers a chance to work locally without resorting to localhost, 127.0.0.1, or tricky hacks, like editing the /etc/hosts file.

These magic DNS servers might be able to help because they can also be used with public IP addresses. That means, they can be used to assign a valid domain to a public IP in situations where those addresses would not normally have one.

Now, let’s go off topic for a moment. Take notice on your Internet connection. As you may know, most broadband providers are rolling out IPv6 to their customers. This is becoming a common practice as part of the IPv6 worldwide adoption. If that’s the case for you, it’s very likely that all of the most modern devices connected to your home Wi-Fi already have a public IPv6 address. Smartphones, laptops, tablets, desktop computers… Yes! In some creepy way, they are ALL available online, reachable from anywhere in the world where IPv6 is routable.

So, go ahead and check out the IP addresses you have in your network interface.

If you see some inet6 address with global scope, that means you are hot, online, right now!

If you’ve already put 2 and 2 together, you’ve realized you have the pieces you need to create a Let’s Encrypt certificate:

Public IP address + Valid domain name that resolves to that address

Here’s an example:

$ nslookup 2800-3f0-4001-810-0-0-0-200e.sslip.io
Server:         208.67.220.220
Address:        208.67.220.220#53

Non-authoritative answer:
Name:   2800-3f0-4001-810-0-0-0-200e.sslip.io
Address: 2800:3f0:4001:810::200e

Note that the IPv6 address needs to be cleaned up in order to be used with sslip.io. Following IPv6 rules, expand :: into its respective zeroes. Then, replace every colon (:) with a dash (-). Repeated zeroes can still be compacted into a single digit.

Now you can use certbot to emit a valid certificate. If you don’t have certbot installed, you can easily run it using the official Docker image.

docker run --rm -it --name certbot --network host \
  -v $(pwd)/data:/etc/letsencrypt \
  certbot/certbot \
  --test-cert --dry-run \
  --standalone \
  certonly --agree-tos \
  -m "myemail@my.domain.com" \
  -d <FORMATTED-IPv6>.sslip.io

The above will run certbot in dry-run mode using Let’s Encrypt staging endpoint. It won’t create a certificate, but it will test the waters and tell you if it’s possible.

To go ahead and create a real certificate, remove the --test-cert and --dry-run parameters.

The local volume directory ./data will contain everything generated inside /etc/letsencrypt, including certbot configuration files and the certificate. To renew it, change certonly to renew, keeping the same ./data directory.

This is pretty much the same task that would be automated somewhere in a real server with public IP addresses, except you can do this manually using a valid domain that points to your local machine.

Installing the certificate is still up to you. That depends on the software, platform and frameworks you are using.

But now, you can test even more scenarios with a valid SSL certificate 😉

Some caveats to remember

As usual, for Let’s Encrypt to work, you still need a public IP address, either v4 or v6. This hint about using IPv6 comes from the fact that its adoption has been increasing worldwide. So, chances are that you already have one by now.

This same trick could also come in handy for your home brew projects, like Raspberry Pi boxes. But in any case, you might be out of luck if your broadband provider hasn’t rolled out IPv6 in your area yet.

By using your IPv6 address, you must also make sure that the software you are running — or developing on — supports IPv6. Aside from its clients resolving names on any public DNS server, the server software itself must also be able to bind to an IPv6 address in order for you to use the <IPv6>.sslip.io domain. Otherwise, the certificate will be valid, but useless.

You might also be out of luck if your local machine is behind a corporate network that does not support or provide public IPv6 addresses. Corporate policies will likely not allow you to have one to begin with.

Either way, if you happen to be on the IPv6 network, you might also want to check your local firewall rules to make sure they allow incoming connections to port 80, at least temporarily.

Let’s get something straight for beginners: A Container is NOT a Virtual Machine

UPDATE 2020/Jan/29: As pointed out by some of the feedback, the term Virtual Machine in this article refers specifically to full x86/x64 virtualization, as described in the current Wikipedia article. It relates to the use of hypervisor and similar technologies to emulate an entire physical machine in software. Please, be aware of this reference while reading, so it is not confused with other types of virtual machines, such as JVM, .NET or interpreted programming language environments.

I’ve been working with Docker, containers, and Kubernetes for over three years now. And, from my perspective, I managed to catch up with this new trend just before it picked up full steam in most developer forums. I’ll admit, it took me more than a few months to understand what a container actually is and how it works.

If you’ve been working at software operations and infrastructure for quite sometime, and by any chance you are just now beginning to catch up, do not be scared. You have A LOT to take in. It does take effort and getting used to it. I remember the feeling, the confusion, the will to give up and go back to provisioning stuff the old fashioned way. I distinctly remember the will to find a nice blog post describing things in a simple way, without making so many assumptions. By now, I’m pretty sure some folks at /r/docker are getting used to watching thread, after thread, after thread of people rambling about their frustration. They need to migrate a full stack to containers and nothing seems to make sense.

So, I decided to write up this simple, quick introduction to welcome beginners into a new era. I will try to uncover some of the magic behind containers, so you don’t feel so lost in the dark. It serves as an introduction to containers before you are introduced to Docker. Something that I feel is missing from most tutorials and guides.

Hopefully, it’ll help you deal with the frustration, clear some of the most basic concepts and pave the way for a better experience with Docker, Kubernetes and everything else.

First things first

If you work with Linux, the basic idea is not really hard to grasp. I wish I had someone tell me this from the beginning:

To understand Linux Containers, you should first understand what makes a Linux Distribution.

Said me to myself when I finally got it

Ubuntu, CentOS, Arch, Alpine, FreeBSD, OpenBSD… We each have our favorite. But, whatever flavor you love, they all have one important thing in common: a Linux Kernel. Making a new Linux Distribution almost never means writing your own kernel from scratch. There already exists a very good one driven by a strong community. For the most part, you just take it, compile it, and bundle it with other stuff to create your distribution.

Inside every common Linux Distro, you will find basically the same types of components grouped into directories in the filesystem:

  • /boot – The kernel, along with whatever it needs to be bootstrapped.
  • /bin – Basic program binaries like cp, ls, cat, grep, echo…
  • /sbin – Program binaries reserved for the root.
  • /etc – System wide configuration files.
  • /lib – System wide libraries.
  • /usr – User installed software, their binaries and libraries.
  • /opt – Proprietary software that won’t follow the above directory structure.
  • /home – User files

Of course, there’s more to that structure, variations and more directories. But that is the basic overview. The cherry on top is a Package Manager so that users can install and manage additional software: dpkg, apt, yum, synaptic, pacman, zypper, rpm… One is enough, so take your pick.

Bundle all that into an ISO image that boots as an installer program, and voilà! You’ll have yourself a working Linux Distribution.

Remember the basics, how programs work

When you run a program, a copy of it goes to RAM and becomes a process managed by the kernel. From there, it expects all of its dependencies to be in place and readily accessible. Among various things, it will usually:

  • Load configuration files from /etc
  • Load libraries from directories like /lib or /usr/lib
  • Write data to /var/some/directory

As long as everything is in place, exactly as expected, a process will run happily ever after.

So, what’s the problem?

Dedicated servers typically run a small number of dedicated processes. For example, in order to host a WordPress blog, a single Linux host can be easily configured with a LAMP stack: MySQL, Apache, and PHP packages installed.

But… what if you need to host more than one WordPress installation? What if each one is required to have their own MySQL instance? Let’s keep going… What if you need to deploy older stacks that require different PHP versions? Different modules? Conflicting libraries? Binaries compiled with different flags and modules?

We are used to solving this problem very bluntly: increase the cost and pay for more resources. The standard response to complex requirements has been the same for so many decades:

We just can’t run everything in one host. Either give me more hosts or create more Virtual Machines. We need to keep things ISOLATED!

Isolation! That’s the key word here

From very early on, the Linux community had been trying to find ways to isolate running processes to avoid dependency conflicts and improve security. Solutions like chroot and jails were notable foreshadows of what came to be known as Linux Containers (LXC). Those were early attempts at process isolation. While chroot and jails were popular and relatively easy, they lacked advanced features. The complexity of LXC, on the other hand, made it difficult for wide adoption.

Up until now, the traditional way of isolating services with security and quality guarantees has meant mostly only one thing: running different services in different hosts. Each with its own Linux installation and dedicated kernel.

The kernel has evolved, and most people never even noticed

For quite some time now, the Linux Kernel has been growing with new exciting features. Today, several different ways to isolate processes have been baked into the kernel itself and are quite ready for production — control groups, namespaces, virtual network interfaces… all kinds of interesting features are there. LXC was a first real attempt to harness those features, but it failed to keep things simple.

Putting it the simplest way possible:

Creating a container means running a Linux process, much like any other, except with very strong isolation, the likes of which no one had ever seen before.

In practice, it means:

  • Create a directory dedicated to your application.
  • Place the application binary, along with everything it needs, inside that directory: dependent libraries, configuration files, data directories…
  • Spawn the application process asking the kernel to isolate EVERYTHING, giving it restrictions like:
    • Its own user space, including access to another less privileged root user, and no visibility or UID/GID conflicts with other users already created outside the container.
    • Its own filesystem structure, with the most important parts (like /etc/hosts) as read-only, even for the container’s root user.
    • Its own process space, with no visibility to any other process and PIDs running in the same kernel.
    • Its own network interface where it can have its own IP and not worry about conflicting ports.
    • Limits to how much time it can spend consuming CPU cycles.
    • Limits to how much memory it can use.

Think of it as chroot or jails on steroids.

You can literally cram an entire different Linux Distribution inside the container directory. A process running inside the container isolation shares the same kernel as other processes, but it can easily think that it’s running completely alone and that it’s part of an entirely different operating system.

If it walks like Alpine, and it quacks like Alpine… Well, I guess I REALLY AM running in the Alpine OS!

Says the process running in a container sharing the kernel bootstrapped by an Ubuntu host.

Containers start much faster than Virtual Machines because they are not bootstrapping a different kernel into a new memory space, along with every other process a full operating system needs. They are simply spawning a new process in the same kernel. The isolation is what makes that process special.

And now, you are ready for Docker

Docker came along as a bold rewrite and recycling of LXC. It completely re-imagined how to create, manage and distribute containers. It made things much, much, much easier, especially at large scale.

Instead of manually creating all of that structure, you simply need dockerd installed and running as a normal system daemon. Container can be created by writing intuitive Dockerfiles, compressed into tarballs and easily ported to other hosts. Under the hood, Docker makes use of OverlayFS to share and merge multiple directory layers into a single view inside the container. It’s a powerful tool that makes Docker containers so versatile.

I’m sure you will find many people listing all the advantages (and disadvantages) of running containerized applications. But the most important, IMHO, is automation. Infrastructure provisioning related to each application becomes code. Code that you write, that can be committed to VCS, traced, shared and integrated with other tools into a pipeline. It’s a bold new way of thinking about infrastructure. It changes a lot. But at large scale, full automation from development, to testing, to production becomes a concrete reality.

Don’t worry! There’s a lot more to learn. But, hopefully, this introduction has given you a foothold so you can dive into the official docs with more confidence: https://docs.docker.com

God speed, and best of luck!

Como customizar uma imagem ubuntu-cloud usando o cloud-init no VirtualBox

Uma versão prática deste exemplo pode ser encontrada no Gitlab:

🔗 https://gitlab.com/juliohm1978/cloud-init-example

Documentação do cloud-init: 🔗 https://cloudinit.readthedocs.io/en/latest/

  • Importar a imagem para o VirtualBox

    No VirtualBox, clique no botão de importação.

    Selecione a imagem OVA que vocẽ baixou.

    Alguns parâmetros da VM podem ser alterados neste momento. Não se esqueça de escolher no Mac Address Policy: Generate new MAC address for all network adapters.

  • Configurar a porta serial do console

    Devido a um problema na configuração desta imagem, ela não consegue iniciar sem a existência de uma porta serial para o console principal.

    Launch Pad Bug #1573095

    Sem a portal serial, a imagem trava durante o boot e não responde. Para contornar o problema, configure a porta serial enviando seu conteúdo para qualquer arquivo temporário — ex: /tmp/serial.log.

    DICA: Você pode visualizar o conteúdo do console em outro terminal para ver informações detalhadas do boot do sistema operacional e a execução do cloud-init.

    tail -F /tmp/serial.log
    

  • Prepare uma imagem ISO com as suas customizações

    Instale as ferramentas do cloud-init.

    sudo apt install cloud-image-utils
    

    Crie os arquivos meta-data.yaml, network-config.yaml e user-data.yaml.

    meta-data.yaml

    dsmode: local
    

    network-config.yaml

    version: 2
    ethernets:
      enp0s3:
        dhcp4: true
    

    user-data.yaml

    #cloud-config
    system_info:
      default_user:
        name: user
        gecos: user
        groups: [wheel, adm]
        sudo: ["ALL=(ALL:ALL) ALL"]
        shell: /bin/bash
    password: user
    ssh_pwauth: true
    chpasswd:
      expire: false
    
    

    Crie a imagem iso usando o comando cloud-localds.

    cloud-localds -v cloud-init.iso --network-config=network-config.yaml user-data.yaml meta-data.yaml
    
  • Anexar a ISO à VM

    Nas configurações da VM, adicione uma unidade cdrom na controladora IDE.

    Adicione e escolha a ISO que você acabou de criar.

  • Configure as placas de rede da VM

    Para este teste simplificado, configure apenas uma placa de rede com NAT.

  • Faça um snapshot da VM zerada

    Antes de iniciar qualquer atividade, faça um snapshot da VM neste momento.

    É importante fazer o snapshot ANTES do primeiro boot. Assim vocẽ poderá retornar ao ponto de partida para realizar testes com diferentes configurações no user-data.

  • Iniciar a VM

    Faça o boot da imagem e confira no /tmp/serial.log o resultado. Mensagens do cloud-init devem aparecer durante o andamento.

    Ao final, você poderá entrar na VM com usuário user e senha user.

  • Customize ainda mais

    Volte o snapshot inicial quantas vezes quiser para testar configurações mais avançadas.

    LEMBRE-SE❗: Sempre que modificar os arquivos Yaml (user-data, network-config ou meta-data) é preciso recriar a imagem ISO.

    Confira exemplos mais completos na documentação oficial.

    🔗 https://cloudinit.readthedocs.io/en/latest/topics/examples.html

    DICA: Para poder acessar a VM via SSH, será preciso adicionar outra placa de rede com Host-Only Network.

Como instalar o Docker

Versão 19.03.5.

Modo mais rápido

Um script de instalação fica hospedado em https://get.docker.com. Ele pode ser usado na maioria das plataformas Linux.

curl -Ls https://get.docker.com > instalador.sh

## confira o conteúdo de instalador.sh antes de continuar

sudo sh instalador.sh

Em diferentes plataformas

Informações detalhadas sobre a instalação em cada plataforma em

🔗 https://docs.docker.com/get-docker/

Por exemplo, como instalar no Ubuntu:

🔗 https://docs.docker.com/install/linux/docker-ce/ubuntu/

Pós instalação

Por questões de segurança, seu usuário não terá permissões para criar ou executar contêineres.

O comando docker pode ser executado com sudo.

sudo docker [build|create|run...]

Para dar permissões ao seu usuário, basta incluí-lo no grupo docker.

CUIDADO: Qualquer usuário com permissões para criar e executar contêineres pode se comportar como administrador/root do sistema operacional.