visibilityspotshttps://visibilityspots.org/2023-07-27T00:00:00+02:00Linux & Open-Source enthusiast | Scouting | LongboardingNomad spread scheduler2023-07-05T22:00:00+02:002023-07-27T00:00:00+02:00Jantag:visibilityspots.org,2023-07-05:/nomad-spread-scheduler.html<p>I'm maintaining a <a href="../nomad-arm-cluster.html">nomad cluster</a> already a few years now at home, based on some thin clients and a few raspberry pi's.</p>
<p>The workload is growing from uses cases of the <a href="../planespotting.html">plane-spotting</a> services towards a <a href="../dockerized-cloudflared-pi-hole.html">pi-hole</a> setup, <a href="https://github.com/dani-garcia/vaultwarden">vaultwarden</a>, <a href="https://www.home-assistant.io/">homeassistant</a> and many more use cases.</p>
<p>One of the issues I encountered was based on the default <a href="https://www.nomadproject.io/docs/concepts/scheduling/scheduling">scheduling algorithm</a>. Raspberry pi's are not known as the most efficient solution to run a huge workload. Default nomad will schedule new containers on one compute node until the resource limits of that node are consumed and only then will start consuming another node. This …</p><p>I'm maintaining a <a href="../nomad-arm-cluster.html">nomad cluster</a> already a few years now at home, based on some thin clients and a few raspberry pi's.</p>
<p>The workload is growing from uses cases of the <a href="../planespotting.html">plane-spotting</a> services towards a <a href="../dockerized-cloudflared-pi-hole.html">pi-hole</a> setup, <a href="https://github.com/dani-garcia/vaultwarden">vaultwarden</a>, <a href="https://www.home-assistant.io/">homeassistant</a> and many more use cases.</p>
<p>One of the issues I encountered was based on the default <a href="https://www.nomadproject.io/docs/concepts/scheduling/scheduling">scheduling algorithm</a>. Raspberry pi's are not known as the most efficient solution to run a huge workload. Default nomad will schedule new containers on one compute node until the resource limits of that node are consumed and only then will start consuming another node. This algorithm is named bin packing which is used to optimize resource utilization.</p>
<p>Although the idea seems leg-it I encountered a lot of issues due to this behavior where raspberry pi's got overloaded and therefor failing applications. From which one is the used DNS solution at home resulting in some mad family members not being able to surf anymore :D</p>
<p>Luckily there is an alternative algorithm available called the <a href="https://developer.hashicorp.com/nomad/docs/other-specifications/node-pool#scheduler_algorithm">spread algorithm</a> which will spread your workload amongst the available nomad nodes.</p>
<p>First of all you can double check your current configuration by querying the nomad <a href="https://developer.hashicorp.com/nomad/api-docs/operator/scheduler#sample-request">operator API</a></p>
<p>You can also use the <a href="https://developer.hashicorp.com/nomad/api-docs/operator/scheduler#update-scheduler-configuration">operator API</a> to update the scheduler configuration on the fly. But when you bootstrap a new node you need to configure it manually again for that particular node.</p>
<p>The service block also has an option to set a default scheduler;</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/etc/nomad.d/nomad.hcl
server<span class="o">{</span>
<span class="w"> </span>default_scheduler_config<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nv">scheduler_algorithm</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"spread"</span>
<span class="w"> </span><span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
<p>And of course restarting the nomad daemon.</p>
<p>Never since I encountered issues again by overloading one of the pi's and therefor resulting in failing applications on top of it.</p>
<p>An other approach would be to use the spread stanza in your nomad jobs, but this seemed to be a too much of a hassle for my home lab setup;</p>
<ul>
<li><a href="https://www.hashicorp.com/blog/spreads-and-affinites-in-nomad">spreads-and-affinites-in-nomad</a></li>
<li><a href="https://developer.hashicorp.com/nomad/tutorials/advanced-scheduling/spread">advanced-scheduling tutorial</a></li>
</ul>Traefik SSL grading2022-08-20T12:00:00+02:002022-08-25T00:00:00+02:00Jantag:visibilityspots.org,2022-08-20:/traefik-ssl-grading.html<p>Recently I discovered that many of the services I deployed upon my <a href="https://visibilityspots.org/nomad-arm-cluster.html">nomad cluster</a> didn't had the SSL A grading I expected them to have. Somehow I asumed the <a href="https://visibilityspots.org/traefik-nomad-route53.html">traefik letsencrypt</a> implementation got the A rating by default.</p>
<p>After running the <a href="https://github.com/drwetter/testssl.sh">testssl.sh</a> container it turns out they don't;</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>--rm<span class="w"> </span>-ti<span class="w"> </span>drwetter/testssl.sh<span class="w"> </span>domain.org
<span class="w"> </span>Rating<span class="w"> </span>specs<span class="w"> </span><span class="o">(</span>not<span class="w"> </span><span class="nb">complete</span><span class="o">)</span><span class="w"> </span>SSL<span class="w"> </span>Labs<span class="s1">'s '</span>SSL<span class="w"> </span>Server<span class="w"> </span>Rating<span class="w"> </span>Guide<span class="err">'</span><span class="w"> </span><span class="o">(</span>version<span class="w"> </span>2009q<span class="w"> </span>from<span class="w"> </span><span class="m">2020</span>-01-30<span class="o">)</span>
<span class="w"> </span>Specification<span class="w"> </span>documentation<span class="w"> </span>https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide
<span class="w"> </span>Protocol<span class="w"> </span>Support<span class="w"> </span><span class="o">(</span>weighted<span class="o">)</span><span class="w"> </span><span class="m">95</span><span class="w"> </span><span class="o">(</span><span class="m">28</span><span class="o">)</span>
<span class="w"> </span>Key<span class="w"> </span>Exchange<span class="w"> </span><span class="o">(</span>weighted<span class="o">)</span><span class="w"> </span><span class="m">100</span><span class="w"> </span><span class="o">(</span><span class="m">30</span><span class="o">)</span>
<span class="w"> </span>Cipher<span class="w"> </span>Strength<span class="w"> </span><span class="o">(</span>weighted<span class="o">)</span><span class="w"> </span><span class="m">90</span><span class="w"> </span><span class="o">(</span><span class="m">36</span><span class="o">)</span>
<span class="w"> </span>Final<span class="w"> </span>Score …</code></pre></div><p>Recently I discovered that many of the services I deployed upon my <a href="https://visibilityspots.org/nomad-arm-cluster.html">nomad cluster</a> didn't had the SSL A grading I expected them to have. Somehow I asumed the <a href="https://visibilityspots.org/traefik-nomad-route53.html">traefik letsencrypt</a> implementation got the A rating by default.</p>
<p>After running the <a href="https://github.com/drwetter/testssl.sh">testssl.sh</a> container it turns out they don't;</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>--rm<span class="w"> </span>-ti<span class="w"> </span>drwetter/testssl.sh<span class="w"> </span>domain.org
<span class="w"> </span>Rating<span class="w"> </span>specs<span class="w"> </span><span class="o">(</span>not<span class="w"> </span><span class="nb">complete</span><span class="o">)</span><span class="w"> </span>SSL<span class="w"> </span>Labs<span class="s1">'s '</span>SSL<span class="w"> </span>Server<span class="w"> </span>Rating<span class="w"> </span>Guide<span class="err">'</span><span class="w"> </span><span class="o">(</span>version<span class="w"> </span>2009q<span class="w"> </span>from<span class="w"> </span><span class="m">2020</span>-01-30<span class="o">)</span>
<span class="w"> </span>Specification<span class="w"> </span>documentation<span class="w"> </span>https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide
<span class="w"> </span>Protocol<span class="w"> </span>Support<span class="w"> </span><span class="o">(</span>weighted<span class="o">)</span><span class="w"> </span><span class="m">95</span><span class="w"> </span><span class="o">(</span><span class="m">28</span><span class="o">)</span>
<span class="w"> </span>Key<span class="w"> </span>Exchange<span class="w"> </span><span class="o">(</span>weighted<span class="o">)</span><span class="w"> </span><span class="m">100</span><span class="w"> </span><span class="o">(</span><span class="m">30</span><span class="o">)</span>
<span class="w"> </span>Cipher<span class="w"> </span>Strength<span class="w"> </span><span class="o">(</span>weighted<span class="o">)</span><span class="w"> </span><span class="m">90</span><span class="w"> </span><span class="o">(</span><span class="m">36</span><span class="o">)</span>
<span class="w"> </span>Final<span class="w"> </span>Score<span class="w"> </span><span class="m">94</span>
<span class="w"> </span>Overall<span class="w"> </span>Grade<span class="w"> </span>B
<span class="w"> </span>Grade<span class="w"> </span>cap<span class="w"> </span>reasons<span class="w"> </span>Grade<span class="w"> </span>capped<span class="w"> </span>to<span class="w"> </span>B.<span class="w"> </span>TLS<span class="w"> </span><span class="m">1</span>.1<span class="w"> </span>offered
<span class="w"> </span>Grade<span class="w"> </span>capped<span class="w"> </span>to<span class="w"> </span>B.<span class="w"> </span>TLS<span class="w"> </span><span class="m">1</span>.0<span class="w"> </span>offered
<span class="w"> </span>Grade<span class="w"> </span>capped<span class="w"> </span>to<span class="w"> </span>A.<span class="w"> </span>HSTS<span class="w"> </span>is<span class="w"> </span>not<span class="w"> </span>offered
</code></pre></div>
<p>Turns out traefik by default offers TLS 1.0 and 1.1 which are deprecated since 2018. So those needs to be disabled and TLS 1.3 needs to be supported.</p>
<p>So I started looking into the matter to get that A+ rating and found out traefik has some <a href="https://doc.traefik.io/traefik/https/tls/#tls-options">tls options</a> which can be configured to get there. So I configured traefik with those extra configuration parameters.</p>
<div class="highlight"><pre><span></span><code>traefik.toml:
[tls.options]
[tls.options.default]
sniStrict = true
minVersion = "VersionTLS12"
curvePreferences = ["CurveP521", "CurveP384"]
cipherSuites = [
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"
]
[tls.options.mintls13]
minVersion = "VersionTLS13"
</code></pre></div>
<p>This already got me towards an A rating. Obviously it was already a big step towards the aim of the A+ rating hence still not there yet so I digged deeper into the matter and tried to get the missing HSTS headers fixed to get that + rating!</p>
<p>And thanks to the internet, <a href="https://www.simplecto.com/improve-traefik-https-encryption-qualys-ssl-labs-testssl-sh/">simplecto</a> & <a href="https://community.traefik.io/t/setting-hsts-headers/3794">traefik community</a> I found out adding an extra middleware to fix those headers could bring me that extra + rating;</p>
<div class="highlight"><pre><span></span><code>traefik.toml:
[entryPoints.https.http]
middlewares = ["securedheaders"]
tls = "true"
</code></pre></div>
<p>By configuring this middleware through the https entrypoint and since all http traffic is redirected towards the https entrypoint now all the traffic which reaches the traefik proxy instance will get those headers by default;</p>
<div class="highlight"><pre><span></span><code><span class="n">nomad</span><span class="w"> </span><span class="n">traefik</span><span class="w"> </span><span class="n">job</span><span class="w"> </span><span class="n">tags</span><span class="p">;</span>
<span class="w"> </span><span class="s2">"traefik.http.middlewares.securedheaders.headers.forceSTSHeader=true"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"traefik.http.middlewares.securedheaders.headers.STSPreload=true"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"traefik.http.middlewares.securedheaders.headers.ContentTypeNosniff=true"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"traefik.http.middlewares.securedheaders.headers.browserXssFilter=true"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"traefik.http.middlewares.securedheaders.headers.STSIncludeSubdomains=true"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"traefik.http.middlewares.securedheaders.headers.STSSeconds=315360000"</span><span class="p">,</span>
</code></pre></div>
<p>This means that all my traefik configured services get that A+ rating by default from now on!!</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>--rm<span class="w"> </span>-ti<span class="w"> </span>drwetter/testssl.sh<span class="w"> </span>domain.org
Rating<span class="w"> </span>specs<span class="w"> </span><span class="o">(</span>not<span class="w"> </span><span class="nb">complete</span><span class="o">)</span><span class="w"> </span>SSL<span class="w"> </span>Labs<span class="s1">'s '</span>SSL<span class="w"> </span>Server<span class="w"> </span>Rating<span class="w"> </span>Guide<span class="err">'</span><span class="w"> </span><span class="o">(</span>version<span class="w"> </span>2009q<span class="w"> </span>from<span class="w"> </span><span class="m">2020</span>-01-30<span class="o">)</span>
<span class="w"> </span>Specification<span class="w"> </span>documentation<span class="w"> </span>https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide
<span class="w"> </span>Protocol<span class="w"> </span>Support<span class="w"> </span><span class="o">(</span>weighted<span class="o">)</span><span class="w"> </span><span class="m">100</span><span class="w"> </span><span class="o">(</span><span class="m">30</span><span class="o">)</span>
<span class="w"> </span>Key<span class="w"> </span>Exchange<span class="w"> </span><span class="o">(</span>weighted<span class="o">)</span><span class="w"> </span><span class="m">100</span><span class="w"> </span><span class="o">(</span><span class="m">30</span><span class="o">)</span>
<span class="w"> </span>Cipher<span class="w"> </span>Strength<span class="w"> </span><span class="o">(</span>weighted<span class="o">)</span><span class="w"> </span><span class="m">90</span><span class="w"> </span><span class="o">(</span><span class="m">36</span><span class="o">)</span>
<span class="w"> </span>Final<span class="w"> </span>Score<span class="w"> </span><span class="m">96</span>
<span class="w"> </span>Overall<span class="w"> </span>Grade<span class="w"> </span>A+
</code></pre></div>Traefik nomad route53 setup2020-12-27T21:00:00+01:002020-12-27T00:00:00+01:00Jantag:visibilityspots.org,2020-12-27:/traefik-nomad-route53.html<p>I have this <a href="https://nomadproject.io">nomad</a> cluster running on some spare devices for a while now. Serving my <a href="https://visibilityspots.org/planespotting.html">plane spotting</a> setup, <a href="https://visibilityspots.org/dockerized-cloudflared-pi-hole.html">dns</a> setup, mqtt bridge and some other services I experiment with throughout the years. Until today I've always relied on the ip addresses to point my browser and other services towards the different services. For my DNS setup I even had to pin the jobs towards specific hardware using <a href="https://www.nomadproject.io/docs/configuration/client#custom-metadata-network-speed-and-node-class">meta</a> data.</p>
<p>But I've always wanted to implement a proxy in between so I could rely on DNS names instead. This would also increase the flexibility of my DNS setup since for …</p><p>I have this <a href="https://nomadproject.io">nomad</a> cluster running on some spare devices for a while now. Serving my <a href="https://visibilityspots.org/planespotting.html">plane spotting</a> setup, <a href="https://visibilityspots.org/dockerized-cloudflared-pi-hole.html">dns</a> setup, mqtt bridge and some other services I experiment with throughout the years. Until today I've always relied on the ip addresses to point my browser and other services towards the different services. For my DNS setup I even had to pin the jobs towards specific hardware using <a href="https://www.nomadproject.io/docs/configuration/client#custom-metadata-network-speed-and-node-class">meta</a> data.</p>
<p>But I've always wanted to implement a proxy in between so I could rely on DNS names instead. This would also increase the flexibility of my DNS setup since for a couple of months now I figured the proxy I was looking into implemented UDP services too.</p>
<p>This proxy is <a href="https://traefik.io/">traefik</a>, a few years ago <a href="https://twitter.com/emilevauge">Emile Vauge</a> talked about it on a meetup we organized with <a href="https://inuits.eu">Inuits</a> in Prague. Ever since it stood on my todo list to get it implemented on my home lab.</p>
<p>But to be honest over the years traefik gained some interest and grew a lot. Which made it less attractive due to it's increased complexity compared to <a href="https://fabiolb.net/">fabio</a> which works very well in combination with consul.</p>
<p>However by the time I had it up and running it lacked some letsencrypt integration as well as UDP ports and there where some doubts about it's <a href="https://github.com/fabiolb/fabio/issues/735">continuity</a>. Due the lack of time I never got it in the state I wanted it to be and the whole proxy plan got a bit dusted away.</p>
<p>This year I started to teach a group of students into linux using <a href="https://github.com/visibilityspots/nomad-consul-prometheus">nomad consul prometheus</a> as a back bone. So my whole cluster gained some love again and I decided to upgrade it to the next level by implementing traefik!</p>
<p>So I started with a simple setup, a nomad traefik job which uses consul as service discovery and added some tags to the traefik job itself to gather the configuration to add itself to the proxy.</p>
<div class="highlight"><pre><span></span><code><span class="n">job</span><span class="w"> </span><span class="ss">"traefik"</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"global"</span>
<span class="w"> </span><span class="n">datacenters</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">[</span><span class="n">"DC1"</span><span class="o">]</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"service"</span>
<span class="w"> </span><span class="k">group</span><span class="w"> </span><span class="ss">"traefik"</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">service</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"traefik"</span>
<span class="w"> </span><span class="n">tags</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">[</span>
<span class="n"> "traefik.enable=true",</span>
<span class="n"> "traefik.http.routers.dashboard.rule=Host(`traefik.example.org`)",</span>
<span class="n"> "traefik.http.routers.dashboard.service=api@internal",</span>
<span class="n"> "traefik.http.routers.dashboard.entrypoints=http",</span>
<span class="n"> </span><span class="o">]</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="n">task</span><span class="w"> </span><span class="ss">"traefik"</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">driver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"docker"</span>
<span class="w"> </span><span class="n">config</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="nc">image</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"traefik:v2.3.6"</span>
<span class="w"> </span><span class="n">force_pull</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">true</span>
<span class="w"> </span><span class="n">network_mode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"host"</span>
<span class="w"> </span><span class="n">logging</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"journald"</span>
<span class="w"> </span><span class="n">config</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">tag</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"TRAEFIK"</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="n">volumes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">[</span>
<span class="n"> "local/traefik.toml:/etc/traefik/traefik.toml"</span>
<span class="n"> </span><span class="o">]</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="n">template</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">destination</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"local/traefik.toml"</span>
<span class="w"> </span><span class="k">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o"><<</span><span class="n">EOF</span>
<span class="o">[</span><span class="n">globals</span><span class="o">]</span>
<span class="w"> </span><span class="n">sendAnonymousUsage</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">false</span>
<span class="w"> </span><span class="n">checkNewVersion</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">false</span>
<span class="o">[</span><span class="n">entryPoints</span><span class="o">]</span>
<span class="w"> </span><span class="o">[</span><span class="n">entryPoints.http</span><span class="o">]</span>
<span class="w"> </span><span class="n">address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">":80"</span>
<span class="w"> </span><span class="o">[</span><span class="n">api</span><span class="o">]</span>
<span class="w"> </span><span class="n">dashboard</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">true</span>
<span class="w"> </span><span class="n">insecure</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">true</span>
<span class="w"> </span><span class="o">[</span><span class="n">providers.file</span><span class="o">]</span>
<span class="w"> </span><span class="n">filename</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"/etc/traefik/traefik.toml"</span>
<span class="w"> </span><span class="o">[</span><span class="n">providers.consulCatalog</span><span class="o">]</span>
<span class="w"> </span><span class="k">prefix</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"traefik"</span>
<span class="w"> </span><span class="n">exposedByDefault</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">false</span>
<span class="w"> </span><span class="o">[</span><span class="n">providers.consulCatalog.endpoint</span><span class="o">]</span>
<span class="w"> </span><span class="n">address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"http://CONSUL-SERVER:8500"</span>
<span class="w"> </span><span class="n">datacenter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"DC1"</span>
<span class="n">EOF</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="err">}</span>
<span class="err">}</span>
</code></pre></div>
<p>by starting this nomad job I got a first working dashboard after I configured traefik.example.org pointing towards the host where my traefik job was running upon in /etc/hosts.</p>
<p>As a second step I configured some middlewares. <a href="https://doc.traefik.io/traefik/middlewares/basicauth/">BasicAuth</a> to enable authentication for several services so I could maybe expose them to others in the future. And some redirection for http towards https.</p>
<p>To achieve this I added an extra https entrypoint to the template and added the middleware as a tag to the traefik job;</p>
<div class="highlight"><pre><span></span><code><span class="k">[entryPoints]</span>
<span class="w"> </span><span class="k">[entryPoints.http]</span>
<span class="w"> </span><span class="na">address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">":80"</span>
<span class="w"> </span><span class="k">[entryPoints.https]</span>
<span class="w"> </span><span class="na">address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">":443"</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>group "traefik" {
service {
name = "traefik"
tags = [
"traefik.enable=true",
"traefik.http.middlewares.http2https.redirectscheme.scheme=https",
</code></pre></div>
<p>As a start I configured every service separately to redirect its http traffic towards https. But I found out by reading through some issues in the community from an <a href="https://community.traefik.io/t/router-not-showing-up-using-consul-nomad/8770">answer</a> by Kugel that this could also achieved by setting a global redirect;</p>
<div class="highlight"><pre><span></span><code>group<span class="w"> </span><span class="s">"traefik"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span>service<span class="w"> </span><span class="p">{</span>
<span class="w"> </span>name<span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"traefik"</span>
<span class="w"> </span>tags<span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kt">[</span>
<span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="s">"traefik.http.routers.catchall.rule=HostRegexp(`{host:(www\\.)?.+}`)"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"traefik.http.routers.catchall.entrypoints=http"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"traefik.http.routers.catchall.middlewares=http2https"</span><span class="p">,</span>
</code></pre></div>
<p>by issuing this catchall rule all the http incoming traffic will be redirected to their https service done in one place instead of a separate configuration for every service.</p>
<p>I gained some traction and configured letsencrypt, by default the letsencrypt certificateResolver uses the httpChallenge, but I didn't want to forward all http/https traffic from the internet towards my setup.</p>
<p>Also when you want to use a <a href="https://doc.traefik.io/traefik/https/acme/#wildcard-domains">wildcard</a> certificate you'll have to use the <a href="https://doc.traefik.io/traefik/https/acme/#dnschallenge">DNS-01</a> challenge. I looked into this wildcard domain since I hit the rate limitations quite fast. I asked certificates for every sub domain using my routers and forgot to mount the acme.json file. So I went to the internet, found a <a href="https://computerz.solutions/traefik-ssl-wildcard-letsencrypt/">solution</a> for using a wildcard certificate for different services and fixed the acme.json mount.</p>
<p>Since I already got my domains configured in AWS I added a CNAME record for all my sub domains to point to the internal ip of the traefik instance which I pinned to a specific nomad client using metadata.</p>
<p>Luckily there is a <a href="https://doc.traefik.io/traefik/https/acme/#providers">route53</a> provider, which only needs some <a href="https://go-acme.github.io/lego/dns/route53/">configuration</a> in the AWS backend such as the IAM policy and a user from which you grab the ID's and keys to configure as environment variables towards the traefik container;</p>
<div class="highlight"><pre><span></span><code> task "traefik" {
driver = "docker"
env {
AWS_ACCESS_KEY_ID = ""
AWS_SECRET_ACCESS_KEY = ""
AWS_HOSTED_ZONE_ID = ""
AWS_REGION = "eu-west-1"
}
</code></pre></div>
<p>Once the AWS part is done you can configure the certificateResolver part in the traefik.toml static configuration mark the commented caServer to use the staging environment of letsencrypt first before putting it into production to not hit the rate limit too soon ;);</p>
<div class="highlight"><pre><span></span><code><span class="k">[certificatesResolvers.letsencrypt.acme]</span>
<span class="w"> </span><span class="na">email</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"letsencrypt@example.org"</span>
<span class="w"> </span><span class="na">storage</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"/etc/traefik/acme/acme.json"</span>
<span class="w"> </span><span class="c1"># caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"</span>
<span class="w"> </span><span class="k">[certificatesResolvers.letsencrypt.acme.dnsChallenge]</span>
<span class="w"> </span><span class="na">provider</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"route53"</span>
<span class="w"> </span><span class="na">delaybeforecheck</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"0"</span>
</code></pre></div>
<p>and last but not least you'll have to reconfigure the traefik consul tags so the tls will be used;</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="k">group</span><span class="w"> </span><span class="ss">"traefik"</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">service</span><span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"traefik"</span>
<span class="w"> </span><span class="n">tags</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">[</span>
<span class="n"> "traefik.http.routers.dashboard.rule=Host(`traefik.example.org`)",</span>
<span class="n"> "traefik.http.routers.dashboard.service=api@internal",</span>
<span class="n"> "traefik.http.routers.dashboard.entrypoints=https",</span>
<span class="n"> "traefik.http.routers.dashboard.tls=true",</span>
<span class="n"> "traefik.http.routers.dashboard.tls.certResolver=letsencrypt",</span>
<span class="n"> "traefik.http.routers.dashboard.tls.domains[0</span><span class="o">]</span><span class="p">.</span><span class="n">main</span><span class="o">=</span><span class="n">example</span><span class="p">.</span><span class="n">org</span><span class="ss">",</span>
<span class="ss"> "</span><span class="n">traefik</span><span class="p">.</span><span class="n">http</span><span class="p">.</span><span class="n">routers</span><span class="p">.</span><span class="n">dashboard</span><span class="p">.</span><span class="n">tls</span><span class="p">.</span><span class="n">domains</span><span class="o">[</span><span class="n">0</span><span class="o">]</span><span class="p">.</span><span class="n">sans</span><span class="o">=*</span><span class="p">.</span><span class="n">example</span><span class="p">.</span><span class="n">org</span><span class="err">"</span>
<span class="w"> </span><span class="err">]</span>
</code></pre></div>
<p>Additional services on their turn can be configured in way they only have to enable tls and therefore falling back towards the wildcard certificate;</p>
<div class="highlight"><pre><span></span><code> group "helloworld" {
service {
name = "helloworld"
tags = [
"traefik.enable=true",
"traefik.http.routers.helloworld-secure.rule=Host(`helloworld.example.org`)",
"traefik.http.routers.helloworld-secure.entrypoints=https",
"traefik.http.routers.helloworld-secure.tls=true",
"traefik.http.routers.helloworld-secure.middlewares=default-auth"
]
</code></pre></div>
<p>So now letsencrypt only requests a wildcard for my domain with *.example.org as an alternative and all my services are using that wildcard. That way I could keep the letsencrypt communication to a bare minimum for my setup.</p>
<p>Next up was the long waiting UDP feature, therefor I had to configure a separate UDP <a href="https://doc.traefik.io/traefik/routing/entrypoints/">entrypoint</a>;</p>
<div class="highlight"><pre><span></span><code><span class="k">[entryPoints.dns]</span>
<span class="w"> </span><span class="na">address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">":53/udp"</span>
</code></pre></div>
<p>and a TCP <a href="https://doc.traefik.io/traefik/routing/routers/#configuring-udp-routers">router</a> using tags in the <a href="https://visibilityspots.org/dockerized-cloudflared-pi-hole.html">pihole</a> nomad job;</p>
<div class="highlight"><pre><span></span><code> group "pihole" {
service {
name = "pihole"
tags = [
"traefik.enable=true",
"traefik.udp.routers.pihole-dns.entrypoints=dns"
]
</code></pre></div>
<p>That way I don't have to pin the pihole containers anymore to a specific node, traefik will proxy the DNS traffic towards the pihole service dynamically which is again a little step up in my humble home lab :)</p>
<p>And as finishing touch I also have configured the nomad service on every node with some <a href="https://www.nomadproject.io/docs/configuration/consul#tags">consul tags</a> by doing so I'm able to configure a router using tags in the traefik job;</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="ss">"traefik.http.routers.nomad-ui.rule=Host(`nomad.example.org`)"</span><span class="p">,</span>
<span class="w"> </span><span class="ss">"traefik.http.routers.nomad-ui.service=nomad-client@consulcatalog"</span><span class="p">,</span>
<span class="w"> </span><span class="ss">"traefik.http.routers.nomad-ui.entrypoints=https"</span><span class="p">,</span>
<span class="w"> </span><span class="ss">"traefik.http.routers.nomad-ui.tls=true"</span>
</code></pre></div>
<p>nomad.example.org will now be redirected to the nomad dashboard through one of the different clients. Since it's a cluster I will always get the same state and don't have to bother which client to access :)</p>
<p>So I have now a nomad cluster running with containers I have absolutely no clue where they are running upon neither on which port they are listening. Traefik is the piece of software in between me using sub domains to access my services and the actual container running the software.</p>
<p>What I still need to look into is to use this proxy also to configure redirections towards static services like my synology and it's services as well as my router's interface etc. And I need to figure out how I could dynamically redirect consul.example.org towards the consul ui!</p>Plane spotting on a nomad cluster2020-03-31T21:00:00+02:002020-03-31T00:00:00+02:00Jantag:visibilityspots.org,2020-03-31:/planespotting.html<p>Some weeks ago I upgraded my plane spotting setup by moving my antenna to the <a href="https://visibilityspots.org/piaware.html">roof</a>. It was worth every single effort I've made into it. My stats are rocking ever since. Until the corona crisis halted almost every airline to standstill..</p>
<p>It gave me some time to thinker about my setup, and for some weird coincidence <a href="https://github.com/mikenye">Mike</a> did create a series of docker containers like I was thinking about to implement. I have one pi connected to the USB device which captures the ADB radio signals.</p>
<p>But in the current situation it also feeds the flightaware service since I …</p><p>Some weeks ago I upgraded my plane spotting setup by moving my antenna to the <a href="https://visibilityspots.org/piaware.html">roof</a>. It was worth every single effort I've made into it. My stats are rocking ever since. Until the corona crisis halted almost every airline to standstill..</p>
<p>It gave me some time to thinker about my setup, and for some weird coincidence <a href="https://github.com/mikenye">Mike</a> did create a series of docker containers like I was thinking about to implement. I have one pi connected to the USB device which captures the ADB radio signals.</p>
<p>But in the current situation it also feeds the flightaware service since I used the docker container for piaware. So I wanted to split those up.</p>
<p>And lucky for me Mike did some great work by creating a <a href="https://github.com/mikenye/docker-readsb">readsb</a> container which only captures the signals and can be provides them to a port which can be easily consumed by services which feeds the positions towards a specific service.</p>
<p>Mike did a great effort on that part too and crafted several docker images which are able to consume an external beast host to provide the messages towards the services;</p>
<ul>
<li><a href="https://github.com/mikenye/docker-piaware">flightaware</a></li>
<li><a href="https://github.com/mikenye/docker-flightradar24">flightradar24</a></li>
<li><a href="https://github.com/mikenye/docker-adsbexchange">adsbexchange</a></li>
<li><a href="https://github.com/mikenye/docker-radarbox">radarbox</a></li>
<li><a href="https://github.com/mikenye/docker-planefinder">planefinder</a></li>
</ul>
<p>By doing so I could have the pi attached to the USB device dedicated for the readsb container and the other services are consuming other nodes in the nomad cluster to provide the leverage towards upstream. That way I have my setup spread over my whole <a href="../nomad-arm-cluster.html">nomad</a> cluster.</p>
<p>Which makes it totally future proof and made my day during this lockdown.</p>Httpd semaphores2020-03-09T19:00:00+01:002020-03-09T00:00:00+01:00Jantag:visibilityspots.org,2020-03-09:/httpd-semaphores.html<p>Recently we encountered some strange issues with httpd on some of our CentOS 7 machines during my current project.</p>
<p>Through our pipeline we restart httpd several times which sometimes leads to this error:</p>
<div class="highlight"><pre><span></span><code><span class="nl">Apache</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">error</span><span class="o">]</span><span class="w"> </span><span class="p">(</span><span class="mi">28</span><span class="p">)</span><span class="k">No</span><span class="w"> </span><span class="nf">space</span><span class="w"> </span><span class="nf">left</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="n">device</span>
</code></pre></div>
<p>After some research we found out the semaphores were all being used blocking httpd daemon to restart.</p>
<p>The list of semaphores can be fetched by issuing</p>
<div class="highlight"><pre><span></span><code># ipcs -st
------ Semaphore Operation/Change Times --------
semid owner last-op last-changed
753664 apache Not set Mon Feb 17 20:20:47 2020
786433 apache Not set Mon Feb 17 20:20:47 2020
720898 …</code></pre></div><p>Recently we encountered some strange issues with httpd on some of our CentOS 7 machines during my current project.</p>
<p>Through our pipeline we restart httpd several times which sometimes leads to this error:</p>
<div class="highlight"><pre><span></span><code><span class="nl">Apache</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">error</span><span class="o">]</span><span class="w"> </span><span class="p">(</span><span class="mi">28</span><span class="p">)</span><span class="k">No</span><span class="w"> </span><span class="nf">space</span><span class="w"> </span><span class="nf">left</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="n">device</span>
</code></pre></div>
<p>After some research we found out the semaphores were all being used blocking httpd daemon to restart.</p>
<p>The list of semaphores can be fetched by issuing</p>
<div class="highlight"><pre><span></span><code># ipcs -st
------ Semaphore Operation/Change Times --------
semid owner last-op last-changed
753664 apache Not set Mon Feb 17 20:20:47 2020
786433 apache Not set Mon Feb 17 20:20:47 2020
720898 apache Not set Mon Feb 17 20:20:47 2020
819203 apache Not set Mon Feb 17 20:20:47 2020
851972 apache Tue Feb 18 10:04:36 2020 Mon Feb 17 20:20:47 2020
884741 apache Tue Feb 18 10:04:36 2020 Mon Feb 17 20:20:47 2020
1540102 apache Not set Wed Feb 19 22:57:02 2020
1572871 apache Not set Wed Feb 19 22:57:02 2020
1507336 apache Not set Wed Feb 19 22:57:02 2020
1605641 apache Not set Wed Feb 19 22:57:02 2020
1638410 apache Thu Feb 20 11:16:48 2020 Wed Feb 19 22:57:02 2020
1671179 apache Thu Feb 20 11:16:48 2020 Wed Feb 19 22:57:02 2020
3276812 apache Not set Sun Feb 23 20:18:54 2020
3309581 apache Not set Sun Feb 23 20:18:54 2020
3244046 apache Not set Sun Feb 23 20:18:54 2020
3342351 apache Not set Sun Feb 23 20:18:54 2020
3375120 apache Mon Feb 24 11:30:21 2020 Sun Feb 23 20:18:54 2020
3407889 apache Mon Feb 24 11:30:21 2020 Sun Feb 23 20:18:54 2020
3538962 apache Not set Mon Feb 24 11:30:21 2020
</code></pre></div>
<p>Open connections are not cleared while restarting the httpd daemon in our case unfortunately which leads after some time into the error.</p>
<p>We did found out that clearing those semaphores fixed the issue. Initially we did this manually by executing a for loop;</p>
<div class="highlight"><pre><span></span><code># ipcrm sem $(ipcs -s | grep apache | awk '{print$2}')
</code></pre></div>
<p>Obviously we didn't wanted to wait for our alerting or colleagues to shout when the httpd daemon is stuck. So we configured the ExecStopPost parameter of the httpd systemd unit.</p>
<p>This is done by simply adding a drop in configuration file for that unit</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="n">cat</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">systemd</span><span class="o">/</span><span class="k">system</span><span class="o">/</span><span class="n">httpd</span><span class="p">.</span><span class="n">service</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">clean</span><span class="o">-</span><span class="n">semaphores</span><span class="p">.</span><span class="n">conf</span>
<span class="o">[</span><span class="n">Service</span><span class="o">]</span>
<span class="n">ExecStopPost</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">ipcs</span><span class="w"> </span><span class="o">-</span><span class="n">s</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">awk</span><span class="w"> </span><span class="s1">'$3 == "apache" {system("ipcrm sem " $2)}'</span>
</code></pre></div>
<p>This will clean out the left over semaphores when the httpd daemon has been stopped right before it starts again.</p>
<p>references;</p>
<ul>
<li><a href="https://access.redhat.com/solutions/78873">https://access.redhat.com/solutions/78873</a></li>
<li><a href="https://www.crybit.com/semaphores-linux/">https://www.crybit.com/semaphores-linux/</a></li>
</ul>Piaware on a nomad cluster2020-01-31T19:00:00+01:002020-01-31T00:00:00+01:00Jantag:visibilityspots.org,2020-01-31:/piaware.html<p>A couple of years ago I stumbled on the <a href="https://flightaware.com/">flightaware</a> website and figured out you could capture radio signals from an airplane using a DVB-T dongle which can be bought for about EUR 15.</p>
<p>As this really triggered me to start monitoring planes above our head I bought myself such a device, hooked it up to a raspberry pi zero, installed the piaware software and started grabbing the messages and pushing the compiled positions through flightaware.</p>
<p>Through time I managed to upgrade my home lab towards a <a href="../nomad-arm-cluster.html">nomad cluster</a>. So I migrated my piaware setup to a nomad node and …</p><p>A couple of years ago I stumbled on the <a href="https://flightaware.com/">flightaware</a> website and figured out you could capture radio signals from an airplane using a DVB-T dongle which can be bought for about EUR 15.</p>
<p>As this really triggered me to start monitoring planes above our head I bought myself such a device, hooked it up to a raspberry pi zero, installed the piaware software and started grabbing the messages and pushing the compiled positions through flightaware.</p>
<p>Through time I managed to upgrade my home lab towards a <a href="../nomad-arm-cluster.html">nomad cluster</a>. So I migrated my piaware setup to a nomad node and wrote a <a href="http://www.visibilityspots.com/documents/piaware/nomad.hcl">nomad job file</a> to run the docker image.</p>
<p>This was a great achievement in my cluster setup regarding the software. For the hardware I initially used a cheap <a href="https://shop.pimoroni.com/products/dvb-t-dongle-ideal-for-ads-b-real-time-plane-tracking">DVB-T</a> dongle but I wanted to get more planes and positions so I invested into better gear.</p>
<p>After saving enough by skipping the weekly wok days at work I ordered myself a <a href="https://thepihut.com/products/flightaware-1090mhz-ads-b-antenna-66cm-26in">1090MHz ADS-B N-Type</a> antenna in combination with the <a href="https://thepihut.com/products/flightaware-pro-stick-plus-usb-sdr-ads-b-receiver">Flightaware Pro Stick Plus</a>. To attach the two together I also got myself a <a href="https://www.amazon.com/gp/product/B01FVWCKKE/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1">15m antenna RF coaxial cable</a>.</p>
<p>As a first test setup I used a large bamboo pole which I hooked up into the garden as you can see;</p>
<p><img alt="bamboo" src="../../images/piaware/bamboo.jpg"></p>
<p>The pi was placed inside of the garage and connected through the wireless network. Unfortunately I had a bad reception and installed a powerline based AP to have a stronger signal in the garage. But this setup suffered from outages from time to time. Causing the pi to lose connection and therefore many messages got lost :(</p>
<p>Some auto reboots every now and then solved the connectivity issues as a quick 'ducktaped' fix but I wasn't quite happy with it.</p>
<p>The temporary solution of the bamboo stick stood for about a year. Since I didn't wanted (neither was allowed by my girlfriend) to drill a hole through the roof. But I figured out our central heating unit has a fresh air outlet I could use to go to the roof.</p>
<p>So once equipped with a fancy new pole made by my brother I went for it and installed the antenna on the roof and the pi indoors attached with a network cable instead of the wifi;</p>
<p><img alt="roof" src="../../images/piaware/roof.jpg"></p>
<p>I then made some time to update the docker container images I used from <a href="https://github.com/mikenye/docker-piaware">Mike Nye</a> and create some pull requests. Once he merged those I upgraded my nomad job file and running on the latest available stable version of piaware in a container!</p>
<p>The move of the antenna together with a stable connection was a big success seen my stats increased significantly! Still the 'Klitsberg' nearby our home is blocking some signals from that direction unfortunately.. Maybe someday I could set up an antenna on top of that hill who knows.</p>
<p>A few days after I upgraded I figured Mike released a <a href="https://github.com/mikenye/docker-flightradar24">docker-fr24feed</a> container image which works in tandem with the docker-piaware one. So I went for it adjusted the nomad job file and now I'm not only pushing messages towards <a href="https://flightaware.com/adsb/stats/user/visibilityspots#stats-89862">flightaware</a> but also towards <a href="https://www.flightradar24.com/account/feed-stats/?id=18592">flightradar24</a></p>
<p>Another thing I still need to figure out is the adjustment of the <a href="https://discussions.flightaware.com/t/thoughts-on-optimizing-gain/44482/2">gain</a> which could me bring even better statistics as today :)</p>
<p>Also since becoming a business member on both platforms because being a feeder could open up a way to an API I could start using to setup my wemos based <a href="https://blog.squix.org/2016/07/esp8266-based-plane-spotter-how-to.html">plane spotter</a> in the future!</p>Nomad ARM cluster2019-08-28T21:00:00+02:002019-08-28T00:00:00+02:00Jantag:visibilityspots.org,2019-08-28:/nomad-arm-cluster.html<p>as I already mentioned a few times before I have some hands on experiences with <a href="https://nomadproject.io">nomad</a>. A couple of weeks ago I wrote about a <a href="../nomad-local-development.html">local development</a> setup based on nomad.</p>
<p>Since quite some time I do have a thinclient which is running some docker containers through a docker-compose file I use for personal stuff at home. But this thinclient is suffering from all the containers I'm trying to spin up. While over thinking this issue I did realize I have quite some raspberry pi's laying around and figured I could maybe set up a cluster for those containers.</p>
<p>Since …</p><p>as I already mentioned a few times before I have some hands on experiences with <a href="https://nomadproject.io">nomad</a>. A couple of weeks ago I wrote about a <a href="../nomad-local-development.html">local development</a> setup based on nomad.</p>
<p>Since quite some time I do have a thinclient which is running some docker containers through a docker-compose file I use for personal stuff at home. But this thinclient is suffering from all the containers I'm trying to spin up. While over thinking this issue I did realize I have quite some raspberry pi's laying around and figured I could maybe set up a cluster for those containers.</p>
<p>Since my previous contact with nomad I thought I gave it a try and spin up a nomad home lab cluster.</p>
<p>Priorities first I bought myself a cluster <a href="https://nl.aliexpress.com/item/32754385272.html">rack</a> tied it together with a small switch and a power supply.</p>
<p><img alt="rack" src="../../images/nomad-cluster/rack.jpeg"></p>
<p>Next I created a custom ArchLinux ARM image based on the <a href="https://disconnected.systems/blog/raspberry-pi-archlinuxarm-setup/">work</a> of <a href="https://disconnected.systems/">disconnected systems</a> which I very appreciated since it saved me quite some time!</p>
<p>Once the image was created I pushed the SD cards in the pi's and booted them all.</p>
<p>I did decide to use consul to automate the clustering part as described in the nomad <a href="https://www.nomadproject.io/guides/operations/cluster/automatic.html">operations guides</a>. That way new nomad clients and servers are automatically recognized and added to the cluster when registered in consul.</p>
<p>The side effect of this is a chicken and egg issue, while it would have been cool to run consul as a docker container in the nomad cluster it couldn't be used to auto join the cluster since consul needs to be running before nomad. That's the reason I opted to run consul as a system service instead.</p>
<h1>docker</h1>
<p>Since we will use the docker ecosystem to run our containers using nomad the docker daemon needs to be installed configured and ran on the system as explained on the <a href="https://wiki.archlinux.org/index.php/Docker">archwiki</a></p>
<h1>consul</h1>
<p><a href="https://consul.io">Consul</a> is a service networking solution to connect and secure services across any runtime platform and public or private cloud.</p>
<h2>installation</h2>
<p>first things first, installing consul using pacman</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>pacman<span class="w"> </span>-S<span class="w"> </span>consul
</code></pre></div>
<h2>configuration</h2>
<p>after that some configuration is required for all the consul nodes by creating a data directory and give the ownership to the consul user</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>mkdir<span class="w"> </span>/opt/consul
$<span class="w"> </span>sudo<span class="w"> </span>chown<span class="w"> </span>-R<span class="w"> </span>consul:<span class="w"> </span>/opt/consul
</code></pre></div>
<p>next step on the list is the actual configuration of the consul service for a server functionality</p>
<p><strong>/etc/consul.d/config.json</strong></p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="s">"acl_default_policy"</span><span class="p">:</span><span class="w"> </span><span class="s">"allow"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"addresses"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="s">"dns"</span><span class="p">:</span><span class="w"> </span><span class="s">"0.0.0.0"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"grpc"</span><span class="p">:</span><span class="w"> </span><span class="s">"0.0.0.0"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"http"</span><span class="p">:</span><span class="w"> </span><span class="s">"0.0.0.0"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"https"</span><span class="p">:</span><span class="w"> </span><span class="s">"0.0.0.0"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="s">"advertise_addr"</span><span class="p">:</span><span class="w"> </span><span class="s">"IP.ADDRESS.OF.THIS.NODE"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"advertise_addr_wan"</span><span class="p">:</span><span class="w"> </span><span class="s">"IP.ADDRESS.OF.THIS.NODE"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"bind_addr"</span><span class="p">:</span><span class="w"> </span><span class="s">"0.0.0.0"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"bootstrap"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="s">"bootstrap_expect"</span><span class="p">:</span><span class="w"> </span><span class="s">"3"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"client_addr"</span><span class="p">:</span><span class="w"> </span><span class="s">"0.0.0.0"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"data_dir"</span><span class="p">:</span><span class="w"> </span><span class="s">"/opt/consul"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"datacenter"</span><span class="p">:</span><span class="w"> </span><span class="s">"NAME_DC"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"disable_update_check"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="s">"domain"</span><span class="p">:</span><span class="w"> </span><span class="s">"consul"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"enable_script_checks"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="s">"enable_syslog"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="s">"encrypt"</span><span class="p">:</span><span class="w"> </span><span class="s">"ENCRYPTION_KEY"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"log_level"</span><span class="p">:</span><span class="w"> </span><span class="s">"INFO"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"node_name"</span><span class="p">:</span><span class="w"> </span><span class="s">"NODE NAME"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"performance"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="s">"leave_drain_time"</span><span class="p">:</span><span class="w"> </span><span class="s">"5s"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"raft_multiplier"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span>
<span class="w"> </span><span class="s">"rpc_hold_timeout"</span><span class="p">:</span><span class="w"> </span><span class="s">"7s"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="s">"ports"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="s">"dns"</span><span class="p">:</span><span class="w"> </span><span class="mi">8600</span><span class="p">,</span>
<span class="w"> </span><span class="s">"http"</span><span class="p">:</span><span class="w"> </span><span class="mi">8500</span><span class="p">,</span>
<span class="w"> </span><span class="s">"server"</span><span class="p">:</span><span class="w"> </span><span class="mi">8300</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="s">"raft_protocol"</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span>
<span class="w"> </span><span class="s">"retry_interval"</span><span class="p">:</span><span class="w"> </span><span class="s">"30s"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"retry_interval_wan"</span><span class="p">:</span><span class="w"> </span><span class="s">"30s"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"retry_join"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s">"CONSUL_SERVER_IPS"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"CONSUL_SERVER_IPS"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"CONSUL_SERVER_IPS"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="s">"retry_max"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span>
<span class="w"> </span><span class="s">"retry_max_wan"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span>
<span class="w"> </span><span class="s">"server"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="s">"syslog_facility"</span><span class="p">:</span><span class="w"> </span><span class="s">"local0"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"ui"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span>
<span class="p">}</span>
</code></pre></div>
<p>This configuration is based on a 3 server node setup, when adding extra clients without the server role you can reuse this config but remove the line of <strong>bootstrap_expect</strong> and change <strong>server</strong> to false when configuring a consul client only node.</p>
<p>Most of the parameters to configure should be self explaining if not you can always check the <a href="https://www.consul.io/docs/agent/options.html">consul documentation</a> for more detailed explanations.</p>
<p>! The encryption key needs to be generated as explained in the <a href="https://www.consul.io/docs/agent/encryption.html">agent encryption</a> section.</p>
<h2>service</h2>
<div class="highlight"><pre><span></span><code># systemctl start consul.service
# systemctl enable consul.service
</code></pre></div>
<h2>gui</h2>
<p>If everything went well you should be able to access the consul web interface on port 8500 at one of the configured ip addresses.</p>
<p><img alt="consul-gui" src="../../images/nomad-cluster/consul-gui.png"></p>
<h1>nomad</h1>
<p>Finally the last and coolest part in the whole setup, <a href="https://www.nomadproject.io/">nomad</a>; the actual cluster orchestrater piece of software.</p>
<h2>configuration</h2>
<p>There are 3 configuration parts;</p>
<ul>
<li>base</li>
<li>server</li>
<li>client</li>
</ul>
<p>All of them based in /etc/nomad.d/</p>
<p>I began with a 3 server/client node setup but added extra client only nodes afterwards.</p>
<p><strong>/etc/nomad.d/base.hcl</strong></p>
<div class="highlight"><pre><span></span><code><span class="nx">name</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"NODENAME"</span>
<span class="nx">region</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"global"</span>
<span class="nx">datacenter</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"DATACENTERNAME"</span>
<span class="nx">advertise</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"IP.ADDRESS"</span>
<span class="w"> </span><span class="nx">rpc</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"IP.ADDRESS"</span>
<span class="w"> </span><span class="nx">serf</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"IP.ADDRESS"</span>
<span class="p">}</span>
<span class="nx">consul</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nx">The</span><span class="w"> </span><span class="nx">address</span><span class="w"> </span><span class="nx">to</span><span class="w"> </span><span class="nx">the</span><span class="w"> </span><span class="nx">Consul</span><span class="w"> </span><span class="nx">agent</span><span class="p">.</span>
<span class="w"> </span><span class="nx">address</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"localhost:8500"</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nx">The</span><span class="w"> </span><span class="nx">service</span><span class="w"> </span><span class="nx">name</span><span class="w"> </span><span class="nx">to</span><span class="w"> </span><span class="nx">register</span><span class="w"> </span><span class="nx">the</span><span class="w"> </span><span class="nx">server</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="nx">with</span><span class="w"> </span><span class="nx">Consul</span><span class="p">.</span>
<span class="w"> </span><span class="nx">server_service_name</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"nomad-servers"</span>
<span class="w"> </span><span class="nx">client_service_name</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"nomad-clients"</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nx">Enables</span><span class="w"> </span><span class="nx">automatically</span><span class="w"> </span><span class="nx">registering</span><span class="w"> </span><span class="nx">the</span><span class="w"> </span><span class="nx">services</span><span class="p">.</span>
<span class="w"> </span><span class="nx">auto_advertise</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="nx">Enabling</span><span class="w"> </span><span class="nx">the</span><span class="w"> </span><span class="nx">server</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="nx">to</span><span class="w"> </span><span class="nx">bootstrap</span><span class="w"> </span><span class="nx">using</span><span class="w"> </span><span class="nx">Consul</span><span class="p">.</span>
<span class="w"> </span><span class="nx">server_auto_join</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="nx">client_auto_join</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">true</span>
<span class="p">}</span>
<span class="nx">data_dir</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"/opt/nomad"</span>
<span class="nx">log_level</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"INFO"</span>
<span class="nx">enable_syslog</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="kc">true</span>
</code></pre></div>
<p><strong>/etc/nomad.d/server.hcl</strong></p>
<div class="highlight"><pre><span></span><code>server {
enabled = true
raft_protocol = 3
bootstrap_expect = 3
}
</code></pre></div>
<p><strong>/etc/nomad.d/client.hcl</strong></p>
<div class="highlight"><pre><span></span><code>client {
enabled = true
meta {
}
}
plugin "docker" {
config {
allow_caps = [ "ALL" ]
}
}
</code></pre></div>
<h2>service</h2>
<p>a systemd unit file is written which will combine all the previous configuration files to start the nomad services;</p>
<p><strong>/usr/lib/systemd/system/nomad.service</strong></p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span>
<span class="na">Description</span><span class="o">=</span><span class="s">nomad agent</span>
<span class="na">Wants</span><span class="o">=</span><span class="s">basic.target</span>
<span class="na">After</span><span class="o">=</span><span class="s">basic.target network.target consul.service</span>
<span class="k">[Service]</span>
<span class="na">User</span><span class="o">=</span><span class="s">root</span>
<span class="na">Group</span><span class="o">=</span><span class="s">bin</span>
<span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/nomad agent -config=/etc/nomad.d</span>
<span class="na">ExecReload</span><span class="o">=</span><span class="s">/bin/kill -HUP $MAINPID</span>
<span class="na">KillMode</span><span class="o">=</span><span class="s">process</span>
<span class="na">Restart</span><span class="o">=</span><span class="s">on-failure</span>
<span class="na">RestartSec</span><span class="o">=</span><span class="s">42s</span>
<span class="k">[Install]</span>
<span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span>
</code></pre></div>
<p>now start and enable the nomad service so we can access the gui to see if everything works as expected.</p>
<div class="highlight"><pre><span></span><code># systemctl start nomad.service
# systemctl enable nomad.service
</code></pre></div>
<h2>gui</h2>
<p><img alt="nomad-gui" src="../../images/nomad-cluster/nomad-gui.png"></p>
<h1>future</h1>
<p>Since I'm migrating my current stacks I will add new post for each of those services I tackled afterwards. Starting with the DNS topic and a <a href="https://www.vaultproject.io/">vault</a> integration.</p>
<p>Also for the different services and how I achieved to run them on my containerized cluster I will try to keep you in the loop through my blog.</p>
<p>For now I use an NFS share from my synology station configured on all the nodes using <a href="https://wiki.archlinux.org/index.php/Autofs">autofs</a> as shared storage. I do hope to get time once to setup a ceph cluster on those nodes too :D</p>
<p>Another topic I would like to tackle are the job files. Right now those are stored in an NFS share but somehow I would like to have something more advanced to run them somehow..</p>ArchLinux on intel compute stick2019-06-24T19:00:00+02:002019-08-28T00:00:00+02:00Jantag:visibilityspots.org,2019-06-24:/compute-sticks.html<p>A few months ago we moved into a brand new office which was furnished with a dozen of samsung displays. Unfortunately the basic player included in those displays isn't capable to add a webpage/url as content. Since we've setted up a <a href="https://smashing.github.io/">smashing</a> instance to create dashboards for each team this was a huge bummer.</p>
<p>While looking for a stable solution many teams brought their own raspberry pi's, chromecasts, airtame devices to at least be able to show something on the displays in the meanwhile.</p>
<p>Since we already had good experiences with an intel compute stick and an intel NUC …</p><p>A few months ago we moved into a brand new office which was furnished with a dozen of samsung displays. Unfortunately the basic player included in those displays isn't capable to add a webpage/url as content. Since we've setted up a <a href="https://smashing.github.io/">smashing</a> instance to create dashboards for each team this was a huge bummer.</p>
<p>While looking for a stable solution many teams brought their own raspberry pi's, chromecasts, airtame devices to at least be able to show something on the displays in the meanwhile.</p>
<p>Since we already had good experiences with an intel compute stick and an intel NUC we decided to get and configure about 8 compute sticks model STK1A32SC with archlinux running to be able to display our dashboards.</p>
<p>For our stand up corners we went for 3 intel NUC's with some peripherals like a web-cam/keyboard and a jabra device to provide proper communication during the stand-ups.</p>
<p>But back to the compute sticks.</p>
<h1>Initial setup</h1>
<p>since we had to install and configure about 8 sticks we decided to configure a base archlinux setup on one stick and using <a href="https://wiki.archlinux.org/index.php/Dd">dd</a> afterwards to get the others up and running with a basic archlinux stack.</p>
<h2>BIOS</h2>
<p>before we could boot a live usb archlinux distro we had to change the operating system setting in the bios to Android</p>
<div class="highlight"><pre><span></span><code><span class="n">boot</span><span class="w"> </span><span class="n">device</span>
<span class="n">press</span><span class="w"> </span><span class="n">F2</span>
<span class="o">-></span><span class="w"> </span><span class="n">Select</span><span class="w"> </span><span class="n">Operating</span><span class="w"> </span><span class="n">System</span>
<span class="o">-></span><span class="w"> </span><span class="n">Android</span>
</code></pre></div>
<p>once that's done and you've rebooted press F10 to boot from the live USB distro</p>
<h2>basic setup following guide</h2>
<p>use wifi-menu command to connect to your wireless network to get some network connectivity first</p>
<p>following the basic <a href="https://wiki.archlinux.org/index.php/Installation_Guide">installation guide</a> we went for an UEFI setup with a <a href="https://wiki.archlinux.org/index.php/EFI_system_partition#GPT_partitioned_disks">GPT partitioned</a> disk.</p>
<p>and opted for this layout where 20G is reserved for the root partition, 7.6G as var and 1G for swap.</p>
<h4>partition layout</h4>
<div class="highlight"><pre><span></span><code><span class="nv">Device</span><span class="w"> </span><span class="nv">Start</span><span class="w"> </span><span class="k">End</span><span class="w"> </span><span class="nv">Sectors</span><span class="w"> </span><span class="nv">Size</span><span class="w"> </span><span class="nv">Type</span>
<span class="o">/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">mmcblk0p1</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="mi">1050623</span><span class="w"> </span><span class="mi">1048576</span><span class="w"> </span><span class="mi">512</span><span class="nv">M</span><span class="w"> </span><span class="nv">EFI</span><span class="w"> </span><span class="nv">System</span>
<span class="o">/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">mmcblk0p2</span><span class="w"> </span><span class="mi">1050624</span><span class="w"> </span><span class="mi">3147775</span><span class="w"> </span><span class="mi">2097152</span><span class="w"> </span><span class="mi">1</span><span class="nv">G</span><span class="w"> </span><span class="nv">Linux</span><span class="w"> </span><span class="nv">swap</span>
<span class="o">/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">mmcblk0p3</span><span class="w"> </span><span class="mi">3147776</span><span class="w"> </span><span class="mi">45090815</span><span class="w"> </span><span class="mi">41943040</span><span class="w"> </span><span class="mi">20</span><span class="nv">G</span><span class="w"> </span><span class="nv">Linux</span><span class="w"> </span><span class="nv">filesystem</span>
<span class="o">/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">mmcblk0p4</span><span class="w"> </span><span class="mi">45090816</span><span class="w"> </span><span class="mi">61071326</span><span class="w"> </span><span class="mi">15980511</span><span class="w"> </span><span class="mi">7</span>.<span class="mi">6</span><span class="nv">G</span><span class="w"> </span><span class="nv">Linux</span><span class="w"> </span><span class="nv">filesystem</span>
</code></pre></div>
<p>We configured <a href="https://wiki.archlinux.org/index.php/GRUB#Generate_the_main_configuration_file">grub</a> assuming the EFI partition being mounted as boot and chrooted using arch-chroot as being explained in the installation guide.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># grub-install --target=x86_64-efi --efi-directory=boot --bootloader-id=GRUB</span>
</code></pre></div>
<h4>tools</h4>
<p>some tools we preinstalled where the SSH daemon along with an authorized key for a specific user and python to be able to run ansible afterwards.</p>
<p>also we configured the wireless network already using <a href="https://wiki.archlinux.org/index.php/Systemd-networkd">systemd-network</a></p>
<p>This besides the preferred stuff which is described in the installation guide.</p>
<p>when everything is installed through the installation guide using arch-chroot you can go ahead and reboot</p>
<h2>reboot</h2>
<p>when you have rebooted you should now enter the Grub to the installed archlinux distribution based on the disk of the compute stick. Once that's working fine you can go ahead.</p>
<h1>create an image with dd on separate stick</h1>
<p>as soon as you got a working compute stick you can reboot and now use the live usb distro again</p>
<div class="highlight"><pre><span></span><code># wifi-menu
# systemctl start sshd.service
# passwd
</code></pre></div>
<p>now try to ssh into the live distro from another machine so you could unplug the keyboard and use that second USB port to connect an empty USB drive to store the image on.</p>
<p><img alt="copy" src="../../images/compute-sticks/copy.jpeg"></p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">lsblk</span>
<span class="nv">NAME</span><span class="w"> </span><span class="nv">MAJ</span>:<span class="nv">MIN</span><span class="w"> </span><span class="nv">RM</span><span class="w"> </span><span class="nv">SIZE</span><span class="w"> </span><span class="nv">RO</span><span class="w"> </span><span class="nv">TYPE</span><span class="w"> </span><span class="nv">MOUNTPOINT</span>
<span class="nv">loop0</span><span class="w"> </span><span class="mi">7</span>:<span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">500</span>.<span class="mi">8</span><span class="nv">M</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="o">/</span><span class="nv">run</span><span class="o">/</span><span class="nv">archiso</span><span class="o">/</span><span class="nv">sfs</span><span class="o">/</span><span class="nv">airootfs</span>
<span class="nv">sda</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">0</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">14</span>.<span class="mi">5</span><span class="nv">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">disk</span>
├─<span class="nv">sda1</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">1</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">614</span><span class="nv">M</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">part</span><span class="w"> </span><span class="o">/</span><span class="nv">run</span><span class="o">/</span><span class="nv">archiso</span><span class="o">/</span><span class="nv">bootmnt</span>
└─<span class="nv">sda2</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">2</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">64</span><span class="nv">M</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">part</span>
<span class="nv">sdb</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">16</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">28</span>.<span class="mi">9</span><span class="nv">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">disk</span>
└─<span class="nv">sdb1</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">17</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">28</span>.<span class="mi">9</span><span class="nv">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">part</span>
<span class="nv">mmcblk0</span><span class="w"> </span><span class="mi">179</span>:<span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">29</span>.<span class="mi">1</span><span class="nv">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">disk</span>
<span class="nv">mmcblk0boot0</span><span class="w"> </span><span class="mi">179</span>:<span class="mi">8</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">4</span><span class="nv">M</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nv">disk</span>
<span class="nv">mmcblk0boot1</span><span class="w"> </span><span class="mi">179</span>:<span class="mi">16</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">4</span><span class="nv">M</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nv">disk</span>
</code></pre></div>
<p>mount the additional USB drive and start creating an image through a screen session</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">mount</span><span class="w"> </span><span class="o">/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">sdb1</span><span class="w"> </span><span class="o">/</span><span class="nv">mnt</span>
#<span class="w"> </span><span class="nv">screen</span><span class="w"> </span><span class="o">-</span><span class="nv">S</span><span class="w"> </span><span class="nv">image</span><span class="o">-</span><span class="nv">creation</span>
#<span class="w"> </span><span class="nv">dd</span><span class="w"> </span><span class="k">if</span><span class="o">=/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">mmcblk0</span><span class="w"> </span><span class="nv">conv</span><span class="o">=</span><span class="nv">sync</span>,<span class="nv">noerror</span><span class="w"> </span><span class="nv">bs</span><span class="o">=</span><span class="mi">64</span><span class="nv">K</span><span class="w"> </span><span class="nv">status</span><span class="o">=</span><span class="nv">progress</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nv">gzip</span><span class="w"> </span><span class="o">-</span><span class="nv">c</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">/</span><span class="nv">mnt</span><span class="o">/</span><span class="nv">base</span><span class="o">-</span><span class="nv">image</span><span class="o">-</span><span class="nv">dashboards</span>.<span class="nv">img</span>.<span class="nv">gz</span>
</code></pre></div>
<h1>restore image</h1>
<p>to restore the image on a new compute stick you have to boot again in live distro mode and enable wifi + ssh to be able to unplug the keyboard USB port once again</p>
<div class="highlight"><pre><span></span><code># wifi-menu
# systemctl start sshd.service
# passwd
</code></pre></div>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">lsblk</span>
<span class="nv">NAME</span><span class="w"> </span><span class="nv">MAJ</span>:<span class="nv">MIN</span><span class="w"> </span><span class="nv">RM</span><span class="w"> </span><span class="nv">SIZE</span><span class="w"> </span><span class="nv">RO</span><span class="w"> </span><span class="nv">TYPE</span><span class="w"> </span><span class="nv">MOUNTPOINT</span>
<span class="nv">loop0</span><span class="w"> </span><span class="mi">7</span>:<span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">500</span>.<span class="mi">8</span><span class="nv">M</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="o">/</span><span class="nv">run</span><span class="o">/</span><span class="nv">archiso</span><span class="o">/</span><span class="nv">sfs</span><span class="o">/</span><span class="nv">airootfs</span>
<span class="nv">sda</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">0</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">14</span>.<span class="mi">5</span><span class="nv">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">disk</span>
├─<span class="nv">sda1</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">1</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">614</span><span class="nv">M</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">part</span><span class="w"> </span><span class="o">/</span><span class="nv">run</span><span class="o">/</span><span class="nv">archiso</span><span class="o">/</span><span class="nv">bootmnt</span>
└─<span class="nv">sda2</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">2</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">64</span><span class="nv">M</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">part</span>
<span class="nv">sdb</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">16</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">28</span>.<span class="mi">9</span><span class="nv">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">disk</span>
└─<span class="nv">sdb1</span><span class="w"> </span><span class="mi">8</span>:<span class="mi">17</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">28</span>.<span class="mi">9</span><span class="nv">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">part</span>
<span class="nv">mmcblk0</span><span class="w"> </span><span class="mi">179</span>:<span class="mi">0</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">29</span>.<span class="mi">1</span><span class="nv">G</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">disk</span>
<span class="nv">mmcblk0boot0</span><span class="w"> </span><span class="mi">179</span>:<span class="mi">8</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">4</span><span class="nv">M</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nv">disk</span>
<span class="nv">mmcblk0boot1</span><span class="w"> </span><span class="mi">179</span>:<span class="mi">16</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="mi">4</span><span class="nv">M</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nv">disk</span>
</code></pre></div>
<p>next mount the USB drive with the base image on</p>
<div class="highlight"><pre><span></span><code># mount /dev/sdb1 /mnt
# ls /mnt
base-image-dashboards.img.gz lost+found
</code></pre></div>
<p>then start a screen session to unpack the image to the disk of the compute stick</p>
<div class="highlight"><pre><span></span><code># screen -S restore
# gunzip -c /mnt/base-image-dashboards.img.gz | dd of=/dev/mmcblk0 status=progress
# umount -R /mnt
</code></pre></div>
<p>and last but not least initiate grub to install the UEFI partition</p>
<div class="highlight"><pre><span></span><code><span class="c1"># mount /dev/mmcblk0p3 /mnt :(</span>
<span class="c1"># mount /dev/mmcblk0p1 /mnt/boot</span>
<span class="c1"># mount /dev/mmcblk0p4 /mnt/var</span>
<span class="c1"># arch-chroot /mnt</span>
<span class="c1"># grub-install --target=x86_64-efi --efi-directory=boot --bootloader-id=GRUB</span>
<span class="n">Installing</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">x86_64</span><span class="o">-</span><span class="n">efi</span><span class="w"> </span><span class="n">platform</span><span class="o">.</span>
<span class="n">Installation</span><span class="w"> </span><span class="n">finished</span><span class="o">.</span><span class="w"> </span><span class="n">No</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="n">reported</span><span class="o">.</span>
<span class="c1"># exit</span>
<span class="c1"># umount -R /mnt</span>
<span class="c1"># reboot</span>
</code></pre></div>
<p>the compute stick should now boot into the new ArchLinux distribution installed on it's disk and can be configured using ansible.</p>
<h1>kiosk mode</h1>
<p>the main goal of our use case was to show an url. First idea was to use <a href="https://luakit.github.io/">luakit</a> as a browser. But luakit isn't available in the official repositories and isn't able to rotate different tabs.</p>
<p>So we went for chromium which is started <a href="https://wiki.archlinux.org/index.php/Xinit#Starting_applications_without_a_window_manager">without a window-manager</a> and <a href="https://wiki.archlinux.org/index.php/Nodm">nodm</a> to automatically start an x session at boot.</p>
<p>Some quirks we had to resolve where the disabling of the auto restore of chromium by altering the Default/Preference file and setting the values of exit_type to none and exited_cleanly to true after which we made the file read only by making it immutable with the chattr command.</p>
<p><img alt="stack" src="../../images/compute-sticks/stack.jpeg"></p>Nomad local development2019-05-14T19:00:00+02:002019-05-14T00:00:00+02:00Jantag:visibilityspots.org,2019-05-14:/nomad-local-development.html<p>Been using <a href="https://www.nomadproject.io/">nomad</a> for a few years now at the customer I got the question couple of weeks ago from some <a href="https://inuits.eu/team/">colleagues</a> if nomad could be used to replace docker-compose.</p>
<p>The reason behind the question is mainly to not vendor lock yourself for local development with the whole docker eco system.</p>
<p>Since I like a certain level of challenge and I do believe it's a valid use case I gave it a try.</p>
<p>This resulted in a <a href="https://github.com/visibilityspots/nomad-local-development">nomad-local-development</a> repository. The 2 major hurdles to take where to use one file like docker-compose.yml and dns resolving between the containers.</p>
<p>The …</p><p>Been using <a href="https://www.nomadproject.io/">nomad</a> for a few years now at the customer I got the question couple of weeks ago from some <a href="https://inuits.eu/team/">colleagues</a> if nomad could be used to replace docker-compose.</p>
<p>The reason behind the question is mainly to not vendor lock yourself for local development with the whole docker eco system.</p>
<p>Since I like a certain level of challenge and I do believe it's a valid use case I gave it a try.</p>
<p>This resulted in a <a href="https://github.com/visibilityspots/nomad-local-development">nomad-local-development</a> repository. The 2 major hurdles to take where to use one file like docker-compose.yml and dns resolving between the containers.</p>
<p>The first one was a rather easy one to achieve. By grouping multiple tasks into a group you can use one job file to start multiple containers. This however doesn't seems to be best practice in the field but for our use case this works quite well.</p>
<p>Resolving DNS between the different containers was a bit harder to tackle in regard to docker-compose where it's working out of the box. To tackle this in nomad one could use <a href="https://www.consul.io/">consul</a> as a service discovery system. Consul comes by default with a <a href="https://www.consul.io/docs/agent/dns.html">DNS interface</a> which can be used to resolve the registered tasks by nomad.</p>
<p>By combing this interface with <a href="https://learn.hashicorp.com/consul/security-networking/forwarding#dnsmasq-setup">DNS forwarding</a> using <a href="http://www.thekelleys.org.uk/dnsmasq/doc.html">dnsmasq</a> this can be solved in a rather proper manner in this matter. When using a default nomad docker task the container will be registered with his nomad client IP in consul.</p>
<p>To get around this you could use the parameter <code>address_mode = "driver"</code> in the service. That way the docker internal ip will be registered and used to resolve when using the consul DNS.</p>
<p>I used <a href="https://www.vagrantup.com/">vagrant</a> to create a sandbox for this test case to fiddle around the nomad ecosystem. How to use the setup is described in the <a href="https://github.com/visibilityspots/nomad-local-development/blob/master/README.md">README</a></p>
<p>Feel free to share your thoughts idea's and suggestions!</p>BIOS upgrade lenovo archlinux2019-05-04T23:00:00+02:002019-05-04T00:00:00+02:00Jantag:visibilityspots.org,2019-05-04:/bios-upgrade-lenovo.html<p>I got some issues with my wired connection lately that the speed wasn't negotiated correctly and it felt back to 10Mb/s as default.</p>
<p>Did some troubleshooting by eliminating various network devices, restarting them but the results didn't satisfy. Being completely random when and when not auto negotiated.</p>
<p>Before becoming insane I decided to update the bios of my machine (being a lenovo T460s).</p>
<p>I did this already in the past and talked about it even on one of our monthly last Friday's at <a href="https://inuits.eu">work</a>. So I was quite sure I had something written about it for future reference but …</p><p>I got some issues with my wired connection lately that the speed wasn't negotiated correctly and it felt back to 10Mb/s as default.</p>
<p>Did some troubleshooting by eliminating various network devices, restarting them but the results didn't satisfy. Being completely random when and when not auto negotiated.</p>
<p>Before becoming insane I decided to update the bios of my machine (being a lenovo T460s).</p>
<p>I did this already in the past and talked about it even on one of our monthly last Friday's at <a href="https://inuits.eu">work</a>. So I was quite sure I had something written about it for future reference but I couldn't find it anymore.</p>
<p>So I decided to write a post how I did it this time so I could refer to it for myself in the future since I don't do this regularly..</p>
<p>First of all you have to find out your serial number so we can download the latest bios from lenovo.</p>
<p>By using <a href="https://www.archlinux.org/packages/extra/x86_64/dmidecode/">dmidecode</a> this can be done through your terminal</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>dmidecode<span class="w"> </span>-s<span class="w"> </span>system-serial-number
</code></pre></div>
<p>This serial number can be used on the lenovo <a href="https://support.lenovo.com/be/en">support</a> site, where you should find the BIOS Update (Bootable CD) iso in the list of Drivers & Software.</p>
<p>Once downloaded we will use <a href="https://aur.archlinux.org/packages/geteltorito/">geteltorito</a> script as being described on the <a href="https://wiki.archlinux.org/index.php/Flashing_BIOS_from_Linux#Bootable_optical_disk_emulation">ArchWiki</a> to extract the iso into an image file which we can copy onto an USB stick</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>geteltorito.pl<span class="w"> </span>-o<span class="w"> </span><image>.img<span class="w"> </span><image>.iso
$<span class="w"> </span>lsblk<span class="w"> </span>-l
$<span class="w"> </span>sudo<span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span><image>.img<span class="w"> </span><span class="nv">of</span><span class="o">=</span><destination><span class="w"> </span><span class="nv">bs</span><span class="o">=</span>512K
</code></pre></div>
<p>This USB device can now be used to boot from (during boot process press ENTER followed by F12 where you can select the USB device)</p>
<p>It's advised to connect your power supply and your battery has been charged about 80-100% before upgrading your BIOS.</p>
<p>After I had upgraded my BIOS my speed is negotiated correctly. I'm not sure if upgrading did the trick since I rebooted all the network devices where I had this issue but it works now and I have an upgraded BIOS so I'm a happy surfer again. :)</p>
<p>Some references:</p>
<ul>
<li><a href="https://www.cyberciti.biz/faq/update-lenovo-bios-from-linux-usb-stick-pen/">https://www.cyberciti.biz/faq/update-lenovo-bios-from-linux-usb-stick-pen/</a></li>
<li><a href="https://wiki.archlinux.org/index.php/Flashing_BIOS_from_Linux">https://wiki.archlinux.org/index.php/Flashing_BIOS_from_Linux</a></li>
<li><a href="https://workaround.org/article/updating-the-bios-on-lenovo-laptops-from-linux-using-a-usb-flash-stick/">https://workaround.org/article/updating-the-bios-on-lenovo-laptops-from-linux-using-a-usb-flash-stick/</a></li>
</ul>Archlinux ARM pi zero cups network print server2018-12-26T19:00:00+01:002021-03-18T00:00:00+01:00Jantag:visibilityspots.org,2018-12-26:/print-server.html<h2>Printing</h2>
<p>Probably like many amongst us the time of the Christmas holidays is perfect to get some IT related stuff back on track. I used to have a print server setup which got broken over time and I didn't found the energy to invest time into fixing it. But the pressure became higher and higher.
111
From both my wife and daughter, especially during the holidays where the wife want to use it to print out tickets and the daughter want to print out color plates..</p>
<p>So during one of the evenings I pulled myself together and installed <a href="https://archlinuxarm.org/">ArchLinux ARM …</a></p><h2>Printing</h2>
<p>Probably like many amongst us the time of the Christmas holidays is perfect to get some IT related stuff back on track. I used to have a print server setup which got broken over time and I didn't found the energy to invest time into fixing it. But the pressure became higher and higher.
111
From both my wife and daughter, especially during the holidays where the wife want to use it to print out tickets and the daughter want to print out color plates..</p>
<p>So during one of the evenings I pulled myself together and installed <a href="https://archlinuxarm.org/">ArchLinux ARM</a> on a pi zero w and went through the following configuration.</p>
<p>First off all install and configure cups on the pi zero following the <a href="https://wiki.archlinux.org/index.php/CUPS">Arch wiki</a></p>
<p>Do not configure any printer yet, only install cups. When following the guide you can also set cups to be <a href="https://wiki.archlinux.org/index.php/CUPS/Printer_sharing#Remote_administration">available</a> on our local network through the browser on port 631;</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">vim</span><span class="w"> </span><span class="o">/</span><span class="nv">etc</span><span class="o">/</span><span class="nv">cups</span><span class="o">/</span><span class="nv">cupsd</span>.<span class="nv">conf</span>
#<span class="w"> </span><span class="nv">Only</span><span class="w"> </span><span class="nv">listen</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">connections</span><span class="w"> </span><span class="nv">from</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">local</span><span class="w"> </span><span class="nv">machine</span>.
#<span class="nv">Listen</span><span class="w"> </span><span class="nv">localhost</span>:<span class="mi">631</span>
<span class="nv">Port</span><span class="w"> </span><span class="mi">631</span>
<span class="nv">Listen</span><span class="w"> </span><span class="o">/</span><span class="nv">run</span><span class="o">/</span><span class="nv">cups</span><span class="o">/</span><span class="nv">cups</span>.<span class="nv">sock</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span>Restrict<span class="w"> </span>access<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>server...
<span class="nt"><Location</span><span class="w"> </span><span class="nt">/></span>
<span class="w"> </span>Order<span class="w"> </span>allow,deny
<span class="w"> </span>Allow<span class="w"> </span>from<span class="w"> </span>@LOCAL
<span class="nt"></Location></span>
#<span class="w"> </span>Restrict<span class="w"> </span>access<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>admin<span class="w"> </span>pages...
<span class="nt"><Location</span><span class="w"> </span><span class="err">/admin</span><span class="nt">></span>
<span class="w"> </span>Order<span class="w"> </span>allow,deny
<span class="w"> </span>Allow<span class="w"> </span>from<span class="w"> </span>@LOCAL
<span class="nt"></Location></span>
#<span class="w"> </span>Restrict<span class="w"> </span>access<span class="w"> </span>to<span class="w"> </span>configuration<span class="w"> </span>files...
<span class="nt"><Location</span><span class="w"> </span><span class="err">/admin/conf</span><span class="nt">></span>
<span class="w"> </span>AuthType<span class="w"> </span>Default
<span class="w"> </span>Require<span class="w"> </span>user<span class="w"> </span>@SYSTEM
<span class="w"> </span>Order<span class="w"> </span>allow,deny
<span class="w"> </span>Allow<span class="w"> </span>from<span class="w"> </span>@LOCAL
<span class="nt"></Location></span>
</code></pre></div>
<p>Once that's done you can start the cups service and enable it to start at boot;</p>
<div class="highlight"><pre><span></span><code># systemctl start cups.service
# systemctl enable cups.service
</code></pre></div>
<p>Next we will configure the USB connected printers as raw on the pi.</p>
<p>first get your printer USB handle</p>
<div class="highlight"><pre><span></span><code># lpinfo -v | grep usb:
direct usb://Samsung/CLP-310%20Series?serial=################
</code></pre></div>
<p>add a new printer with the found USB handle</p>
<div class="highlight"><pre><span></span><code># lpadmin -p Visibilityspots-LaserJet -v usb://Samsung/CLP-310%20Series?serial=##################
# lpstat -p Visibilityspots-LaserJet -l
printer Visibilityspots-LaserJet disabled since Mon Dec 24 20:51:24 2018 -
reason unknown
</code></pre></div>
<p>enable the printer</p>
<div class="highlight"><pre><span></span><code># cupsenable Visibilityspots-LaserJet
# lpstat -p Visibilityspots-LaserJet -l
printer Visibilityspots-LaserJet is idle. enabled since Mon Dec 24 20:51:54 2018
</code></pre></div>
<p>make the printer accept jobs</p>
<div class="highlight"><pre><span></span><code># cupsaccept Visibilityspots-LaserJet
</code></pre></div>
<p>We do have a server now which can accept print jobs.</p>
<p>But before sending print request we need to configure our clients too. Since we configured a raw printer we need to install the printer drivers on the clients. Depending on you distribution you need to install some specific packages for the specific print drivers for your printer.</p>
<p>I on my ArchLinux I had to install the packages samsung-drivers for my samsung printer and hplib for the hp one.</p>
<p>To get the printers configured I just used the cups web interface on my machine and used the URI <em>http://IP-ADDRESS-PRINT-SERVER-ON-A-PI:631/printers/Visibilityspots-DeskJet</em> in combination with the driver for the specific model.</p>
<p>After that I could easily print the common know print test page which magically printer the tux!!</p>
<h2>Scan</h2>
<p>Besides printing I also wanted to enable the scan option on one of the dives over the network.</p>
<p>This was rather easy as being described on the <a href="https://wiki.archlinux.org/index.php/Sane#Sharing_your_scanner_over_a_network">ArchWiki</a> after I installed both sane and hp-lib on the print-server too.</p>
<p>Do not forget to enable the saned.socket systemd service on your print-server!</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>saned.socket
</code></pre></div>
<p>And now I'm able to put something in the scanner bed and execute a custom scan function I wrote in my zsh config</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>scan<span class="w"> </span>test-image
</code></pre></div>
<p>zsh config:</p>
<div class="highlight"><pre><span></span><code>scan () {
scanimage --device "net:IP-ADDRESS-PRINT-SERVER-ON-A-PI:hpaio:/usb/Deskjet_F4100_series?serial=#################" --resolution 600 -p | pnmtops | ps2pdf - "$1.pdf"
}
</code></pre></div>
<p>which creates a pdf file in the current directory based on the scanned file.</p>
<p>So yipeeyayee I do now have again a working network based print/scan setup without too much effort and still using our old offline printer setups!!</p>
<p>references:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/26329186/creating-a-raw-printer-queue-in-cups-host-and-adding-them-through-cups-client">https://stackoverflow.com/questions/26329186/creating-a-raw-printer-queue-in-cups-host-and-adding-them-through-cups-client</a></li>
<li><a href="https://forum.manjaro.org/t/how-to-set-up-a-remote-printer-which-is-attached-to-a-raspberry-pi-or-any-other-arm-computer/57056y">https://forum.manjaro.org/t/how-to-set-up-a-remote-printer-which-is-attached-to-a-raspberry-pi-or-any-other-arm-computer/57056y</a></li>
</ul>Ansible-playbook archlinux upgrade2018-08-06T19:00:00+02:002018-10-30T00:00:00+01:00Jantag:visibilityspots.org,2018-08-06:/ansible-archlinux-upgrade.html<p>Since a few years now I'm a happy <a href="https://www.archlinux.org/">Archlinux</a> user. I like their <a href="https://wiki.archlinux.org/index.php/Arch_Linux">philosophy</a> which was one of the major points why I made the switch back in the days.</p>
<p>I'm not only using it on my laptop, but do have some devices running at home which are configured with it. From a thin client which I use as a docker node through some raspberry pies running <a href="https://archlinuxarm.org/">ArchlinuxARM</a>.</p>
<p>Since Arch is a rolling update distro there are several updates available throughout the day. To keep on top of them I had to log in on all those devices at least …</p><p>Since a few years now I'm a happy <a href="https://www.archlinux.org/">Archlinux</a> user. I like their <a href="https://wiki.archlinux.org/index.php/Arch_Linux">philosophy</a> which was one of the major points why I made the switch back in the days.</p>
<p>I'm not only using it on my laptop, but do have some devices running at home which are configured with it. From a thin client which I use as a docker node through some raspberry pies running <a href="https://archlinuxarm.org/">ArchlinuxARM</a>.</p>
<p>Since Arch is a rolling update distro there are several updates available throughout the day. To keep on top of them I had to log in on all those devices at least once a day to perform the updates. Experience learned me that let them drifting could lead to some major troubles when only updating after a few weeks.</p>
<p>But it became a time consuming task to keep them all in line. Since <a href="https://www.ansible.com/">ansible</a> is used at the project I'm currently working at it seemed a good idea to write a playbook to update all those devices with only one command. And without having to configure some additional software on all the devices but based on good old SSH.</p>
<p>Ansible already has a default <a href="https://docs.ansible.com/ansible/latest/modules/pacman_module.html">pacman</a> module which can be used for the official repositories. But since a lot of packages I installed are coming from the <a href="https://aur.archlinux.org">AUR</a> I first went with a command execution for <a href="https://github.com/polygamma/aurman">aurman</a>. After some research I found out about <a href="https://github.com/kewlfft/ansible-aur">ansible-aur</a> a bit later so I installed the module and rewrote my playbook so it used the aurman helper.</p>
<p>But only after a few weeks I found out that the developer wasn't really born with an open-source mind as can be seen by his commits <a href="https://github.com/polygamma/aurman/commit/dcb50aa1bf5296dfadbffbe867d3e7e807442397">dcb50aa</a> & <a href="https://github.com/polygamma/aurman/commit/c409feef4c93137c2f0917d8ecdede2d51e06ea9">c409fee</a> so I went for the <a href="https://github.com/Jguer/yay">yay</a> implementation instead.</p>
<p>In the initial phase I used to push my passwords as hashes into the playbook. But when I was about to push the playbook in <a href="https://github.com/visibilityspots/ansible-playbook-archlinux-update">github</a> I figured it wouldn't be a good idea to share that with the public. So I stumbled on <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_vault.html">ansible-vault</a>.</p>
<p>That way I could refer to passwords in an encrypted file in the playbook so I could safely push the playbook to the public. In combination with the parameter <a href="https://docs.ansible.com/ansible/latest/cli/ansible-playbook.html#cmdoption-ansible-playbook-vault-password-file">--vault-password-file</a> I can now run the playbook without interaction for passwords.</p>
<p>And it works great, keeping them all up to date and having a clear output about which packages are updated on which machine. Yet another step closer to that ultimate dream of drinking cocktails on the beach while everything is running automatically in the back!</p>Prometheus export/import2018-06-06T22:00:00+02:002018-06-11T00:00:00+02:00Jantag:visibilityspots.org,2018-06-06:/prometheus-export-import.html<p>bumping into the case where once deployed a full stack application we don't have any direct connection due to no uplink for security reasons.</p>
<p>So we (you too <a href="https://twitter.com/TomVanHumbeeck">@Tom</a>) looked into a way to export the prometheus data into a tar.gz which could be transferred and imported into an instance on our local machine.</p>
<p>After the initial blog post where we created a tar.gz file from the prometheus storage.tsdb.path on the filesystem <a href="https://twitter.com/roidelapluie">@roidelapluie</a> pointed me out about the <a href="https://prometheus.io/docs/prometheus/latest/querying/api/#snapshot">snapshot</a> feature.</p>
<p>So we did a bit of research and came up with this new procedure.</p>
<p>First of …</p><p>bumping into the case where once deployed a full stack application we don't have any direct connection due to no uplink for security reasons.</p>
<p>So we (you too <a href="https://twitter.com/TomVanHumbeeck">@Tom</a>) looked into a way to export the prometheus data into a tar.gz which could be transferred and imported into an instance on our local machine.</p>
<p>After the initial blog post where we created a tar.gz file from the prometheus storage.tsdb.path on the filesystem <a href="https://twitter.com/roidelapluie">@roidelapluie</a> pointed me out about the <a href="https://prometheus.io/docs/prometheus/latest/querying/api/#snapshot">snapshot</a> feature.</p>
<p>So we did a bit of research and came up with this new procedure.</p>
<p>First of all make sure the prometheus service is running with the parameter <strong>--web.enable-admin-api</strong> which is disabled by default. For nomad I created a <a href="https://github.com/visibilityspots/nomad-consul-prometheus/blob/master/nomad/prometheus.hcl">job file</a> which enables this parameter through the args for you.</p>
<p>Once you have the prometheus instance running with the admin api enabled you can use this api to create a snapshot:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>curl<span class="w"> </span>-XPOST<span class="w"> </span>http://localhost:9090/api/v1/admin/tsdb/snapshot
<span class="o">{</span><span class="s2">"status"</span>:<span class="s2">"success"</span>,<span class="s2">"data"</span>:<span class="o">{</span><span class="s2">"name"</span>:<span class="s2">"20180611T130634Z-69ffcdcc60b89e54"</span><span class="o">}}</span>
</code></pre></div>
<p>Next up is to collect the snapshot directory on your local machine and mount it into a fresh prometheus docker container for example.</p>
<div class="highlight"><pre><span></span><code><span class="o">(</span>$<span class="w"> </span>docker<span class="w"> </span>container<span class="w"> </span>list<span class="o">)</span>
<span class="o">(</span>CONTAINER<span class="w"> </span>ID<span class="w"> </span>IMAGE<span class="w"> </span>COMMAND<span class="w"> </span>CREATED<span class="w"> </span>STATUS<span class="w"> </span>PORTS<span class="w"> </span>NAMES<span class="o">)</span>
<span class="o">(</span>e19269f0c82c<span class="w"> </span>cc866859f8df<span class="w"> </span><span class="s2">"/bin/prometheus --c…"</span><span class="w"> </span><span class="m">45</span><span class="w"> </span>minutes<span class="w"> </span>ago<span class="w"> </span>Up<span class="w"> </span><span class="m">45</span><span class="w"> </span>minutes<span class="w"> </span>prometheus-eccbde40-c402-60cf-bee7-04a2e7e77883<span class="o">)</span>
<span class="o">(</span>$<span class="w"> </span>docker<span class="w"> </span>cp<span class="w"> </span>e19269f0c82c:/prometheus/snapshots<span class="w"> </span>/tmp/<span class="o">)</span>
$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>--rm<span class="w"> </span>-p<span class="w"> </span><span class="m">9090</span>:9090<span class="w"> </span>-uroot<span class="w"> </span>-v<span class="w"> </span>/tmp/snapshots/20180611T130634Z-69ffcdcc60b89e54/:/prometheus<span class="w"> </span>prom/prometheus<span class="w"> </span>--config.file<span class="o">=</span>/etc/prometheus/prometheus.yml<span class="w"> </span>--storage.tsdb.path<span class="o">=</span>/prometheus
</code></pre></div>
<p>When you now go to <a href="http://localhost:9090">http://localhost:9090</a> you have the data available from the snapshot and you could start troubleshooting.</p>
<p>For example by starting a grafana container next to this prometheus container and configuring the prometheus one as data source.</p>
<p>That way you could create some dashboards for readability.</p>
<p>references:
- https://www.nomadproject.io/guides/nomad-metrics.html
- https://www.robustperception.io/taking-snapshots-of-prometheus-data/</p>Prometheus consul service discovery2018-06-04T22:00:00+02:002018-06-04T00:00:00+02:00Jantag:visibilityspots.org,2018-06-04:/prometheus-consul.html<p>as published a few months ago I worked out a <a href="http://localhost:8000/jenkins-docker-pipeline.html">dockerized a jenkins farm</a> where both master as slaves are docker containers working together with services like nexus and such. Next to that setup I've dockerized my home setup where services like pi-hole, home-assistant and others are running as docker containers on a thin client I promoted to my home lab.</p>
<p>To have an overview about all those containers and the resources they are consuming I pulled in the git repo of <a href="https://github.com/vegasbrianc/prometheus">Brian Christner</a> which spins up a whole <a href="https://prometheus.io">prometheus</a> stack with some exporters and a grafana instance to visualize …</p><p>as published a few months ago I worked out a <a href="http://localhost:8000/jenkins-docker-pipeline.html">dockerized a jenkins farm</a> where both master as slaves are docker containers working together with services like nexus and such. Next to that setup I've dockerized my home setup where services like pi-hole, home-assistant and others are running as docker containers on a thin client I promoted to my home lab.</p>
<p>To have an overview about all those containers and the resources they are consuming I pulled in the git repo of <a href="https://github.com/vegasbrianc/prometheus">Brian Christner</a> which spins up a whole <a href="https://prometheus.io">prometheus</a> stack with some exporters and a grafana instance to visualize the different aspects of those containers.</p>
<p>The prometheus has been configured manually with hard coded endpoints to scrape in both situations. It works fine but somehow I would have liked prometheus to automatically recognizing the endpoints himself.</p>
<p>Luckily I bumped into <a href="https://www.nomadproject.io">nomad</a> at my current project. Nomad is a tool for managing a cluster of machines and running applications on them which uses <a href="https://www.consul.io/">consul</a> as a key value store. And guess what, prometheus has an integration with consul!</p>
<p>Isn't that just great! By configuring prometheus to find it's endpoints in consul the only line of code is the one to point prometheus where he can find consul.</p>
<div class="highlight"><pre><span></span><code> <span class="k">-</span> job_name: 'self'
consul_sd_configs:
<span class="k">-</span> server: 'localhost:8500'
services: []
</code></pre></div>
<p>But we went a bit further and used tags for that auto discovery. Prometheus will only fetch endpoints which are registered in consul with a certain tag. That way we hold some control in the configuration after all.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="nl">relabel_configs</span><span class="p">:</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nl">source_labels</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">__meta_consul_tags</span><span class="o">]</span>
<span class="w"> </span><span class="nl">regex</span><span class="p">:</span><span class="w"> </span><span class="p">.</span><span class="o">*</span><span class="p">,</span><span class="n">metrics</span><span class="p">,.</span><span class="o">*</span>
<span class="w"> </span><span class="k">action</span><span class="err">:</span><span class="w"> </span><span class="n">keep</span>
</code></pre></div>
<p>Another line is used to rename the endpoints in prometheus by a more human readable one instead of the auto generated one</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nl">source_labels</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">__meta_consul_service</span><span class="o">]</span>
<span class="w"> </span><span class="nl">target_label</span><span class="p">:</span><span class="w"> </span><span class="n">job</span>
</code></pre></div>
<p>And that's about it. With this <a href="https://github.com/visibilityspots/nomad-consul-prometheus/blob/master/prometheus/prometheus.yml">prometheus.yml</a> configuration file services started through nomad with the proper 'metrics' tag are auto discovered by prometheus as target.</p>
<p>To demonstrate this behavior I created a <a href="https://github.com/visibilityspots/nomad-consul-prometheus">github repository</a> based on <a href="https://www.vagrantup.com">vagrant</a> inspired by the <a href="https://www.nomadproject.io/intro/getting-started/install.html">getting started</a> guide of nomad.</p>
<p>Following the <a href="https://github.com/visibilityspots/nomad-consul-prometheus/blob/master/README.md">README</a> a prometheus consul stack is configured and running with 2 exporters you can automatically add to prometheus by starting them through nomad.</p>
<p>It's a pretty cool feeling when the appear and disappear without any manual configuration!</p>
<p>A great reference which cleared my mind during my quest on this topic came from <a href="https://www.robustperception.io/finding-consul-services-to-monitor-with-prometheus/">robust perception</a></p>dockerized DNS over HTTPS using pi-hole through cloudflared proxy-dns2018-04-21T21:00:00+02:002019-02-18T00:00:00+01:00Jantag:visibilityspots.org,2018-04-21:/dockerized-cloudflared-pi-hole.html<p>a few months ago I configured a thin client as my home server to replace the previous <a href="../raspberry-pi.html">raspberry pi</a> setup.</p>
<p>During that migration I moved over all native services within docker containers. One of those services being a <a href="https://pi-hole.net">pi-hole</a> setup to block ad serving domains on dns level and to have a dns cache within our LAN to gain a bit of speed.</p>
<p>It has been running ever since without any issue and worked pretty well.</p>
<p>When cloudflare <a href="https://blog.cloudflare.com/announcing-1111/">announced</a> their fast and privacy based DNS resolver I got a bit intrigued by their DNS over HTTPS feature. Especially since our …</p><p>a few months ago I configured a thin client as my home server to replace the previous <a href="../raspberry-pi.html">raspberry pi</a> setup.</p>
<p>During that migration I moved over all native services within docker containers. One of those services being a <a href="https://pi-hole.net">pi-hole</a> setup to block ad serving domains on dns level and to have a dns cache within our LAN to gain a bit of speed.</p>
<p>It has been running ever since without any issue and worked pretty well.</p>
<p>When cloudflare <a href="https://blog.cloudflare.com/announcing-1111/">announced</a> their fast and privacy based DNS resolver I got a bit intrigued by their DNS over HTTPS feature. Especially since our ISP telenet is using our <a href="http://www.forceflow.be/2016/09/14/aanpassingen-privacybeleid-telenet/">web history</a> for their advertisements too.</p>
<p>So I stumbled on some articles from <a href="https://oliverhough.cloud/blog/configure-pihole-with-dns-over-https/">Oliver Hough</a> and <a href="https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/">Scott Helme</a> that describe how you can combine a <a href="https://developers.cloudflare.com/1.1.1.1/dns-over-https/cloudflared-proxy/">cloudflared proxy-dns</a> with pi-hole to get your dns requests encrypted through HTTPS and still be able to filter out the advertisements.</p>
<p>Since I got everything in docker I configured a <a href="https://hub.docker.com/r/visibilityspots/cloudflared/">cloudflared</a> container automated through <a href="https://travis-ci.org/visibilityspots/dockerfile-cloudflared">travis</a> with <a href="https://github.com/aelsabbahy/goss/tree/master/extras/dgoss">dgoss</a> tests.</p>
<p>I got some inspiration from <a href="https://twitter.com/MaartjeME">maartje</a> who used a <a href="https://github.com/meyskens/docker-cloudflared/blob/master/.travis.yml">matrix</a> to build multiple docker images for different architectures using travis. The main reason behind this was that after I got this setup up and running using this docker-compose file on my x86_64 machine I wanted to run it on a raspberry pi zero w.</p>
<p>For the pihole container I figured out you can easily pass by the custom DNS servers through docker environment variables so no need anymore for a custom pihole docker container to maintain!</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>docker-compose.yml
version:<span class="w"> </span><span class="s2">"3"</span>
services:
<span class="w"> </span>cloudflared:
<span class="w"> </span>container_name:<span class="w"> </span>cloudflared
<span class="w"> </span>image:<span class="w"> </span>visibilityspots/cloudflared:amd64
<span class="w"> </span>restart:<span class="w"> </span>unless-stopped
<span class="w"> </span>networks:
<span class="w"> </span>pihole_net:
<span class="w"> </span>ipv4_address:<span class="w"> </span><span class="m">10</span>.0.0.2
<span class="w"> </span>pi-hole:
<span class="w"> </span>container_name:<span class="w"> </span>pi-hole
<span class="w"> </span>image:<span class="w"> </span>pihole/pihole:v4.2.1_amd64
<span class="w"> </span>restart:<span class="w"> </span>unless-stopped
<span class="w"> </span>ports:
<span class="w"> </span>-<span class="w"> </span><span class="s2">"80:80/tcp"</span>
<span class="w"> </span>-<span class="w"> </span><span class="s2">"53:53/tcp"</span>
<span class="w"> </span>-<span class="w"> </span><span class="s2">"53:53/udp"</span>
<span class="w"> </span>environment:
<span class="w"> </span>-<span class="w"> </span><span class="nv">ServerIP</span><span class="o">=</span><span class="m">10</span>.0.0.3
<span class="w"> </span>-<span class="w"> </span><span class="nv">DNS1</span><span class="o">=</span><span class="s1">'10.0.0.2#5054'</span>
<span class="w"> </span>-<span class="w"> </span><span class="nv">DNS2</span><span class="o">=</span><span class="s1">''</span>
<span class="w"> </span>-<span class="w"> </span><span class="nv">IPv6</span><span class="o">=</span><span class="nb">false</span>
<span class="w"> </span>-<span class="w"> </span><span class="nv">TZ</span><span class="o">=</span>CEST-2
<span class="w"> </span>-<span class="w"> </span><span class="nv">DNSMASQ_LISTENING</span><span class="o">=</span>all
<span class="w"> </span>-<span class="w"> </span><span class="nv">WEBPASSWORD</span><span class="o">=</span>admin
<span class="w"> </span>networks:
<span class="w"> </span>pihole_net:
<span class="w"> </span>ipv4_address:<span class="w"> </span><span class="m">10</span>.0.0.3
<span class="w"> </span>dns:
<span class="w"> </span>-<span class="w"> </span><span class="m">127</span>.0.0.1
<span class="w"> </span>-<span class="w"> </span><span class="m">1</span>.1.1.1
<span class="w"> </span>cap_add<span class="s2">"</span>
<span class="s2"> - NET_ADMIN</span>
<span class="s2">networks:</span>
<span class="s2"> pihole_net:</span>
<span class="s2"> driver: bridge</span>
<span class="s2"> ipam:</span>
<span class="s2"> config:</span>
<span class="s2"> - subnet: 10.0.0.0/29</span>
</code></pre></div>
<p>I remembered this <a href="https://learn.adafruit.com/pi-hole-ad-blocker-with-pi-zero-w">project</a> where a raspberry pi zero W was used together with a tiny display. In the meanwhile I have the DoH cloudflared/pi-hole combination running on such a tiny device using <a href="https://archlinuxarm.org">ArchLinux ARM</a> and ordered the display :D</p>
<p>You can use the same dockerfile on a raspberry pi zero but with other tags for the container images:</p>
<div class="highlight"><pre><span></span><code><span class="n">image</span><span class="o">:</span><span class="w"> </span><span class="n">visibilityspots</span><span class="o">/</span><span class="n">cloudflared</span><span class="o">:</span><span class="n">arm</span>
<span class="n">image</span><span class="o">:</span><span class="w"> </span><span class="n">pihole</span><span class="o">/</span><span class="n">pihole</span><span class="o">:</span><span class="n">v4</span><span class="o">.</span><span class="mi">0</span><span class="n">_armhf</span>
</code></pre></div>
<p>As you can see unfortunately I had to configure static ip's since the dnsmasq config needs the ip address of the cloudflared service. If someone has a better solution to implement it let me know!</p>
<p>I also opted to not store the data. Meaning that when the docker containers are restarted the data is gone.</p>
<p>So when you now bring up those 2 containers:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span>up<span class="w"> </span>-d
Creating<span class="w"> </span>network<span class="w"> </span><span class="s2">"###_pihole_net"</span><span class="w"> </span>with<span class="w"> </span>driver<span class="w"> </span><span class="s2">"bridge"</span>
Creating<span class="w"> </span>pi-hole<span class="w"> </span>...
Creating<span class="w"> </span>cloudflared<span class="w"> </span>...
Creating<span class="w"> </span>pi-hole
Creating<span class="w"> </span>cloudflared<span class="w"> </span>...<span class="w"> </span><span class="k">done</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker-compose<span class="w"> </span>logs<span class="w"> </span>cloudflared
Attaching<span class="w"> </span>to<span class="w"> </span>cloudflared
cloudflared<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="nv">time</span><span class="o">=</span><span class="s2">"2018-04-16T20:01:14Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>info<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Adding DNS upstream"</span><span class="w"> </span><span class="nv">url</span><span class="o">=</span><span class="s2">"https://1.1.1.1/.well-known/dns-query"</span>
cloudflared<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="nv">time</span><span class="o">=</span><span class="s2">"2018-04-16T20:01:14Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>info<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Adding DNS upstream"</span><span class="w"> </span><span class="nv">url</span><span class="o">=</span><span class="s2">"https://1.0.0.1/.well-known/dns-query"</span>
cloudflared<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="nv">time</span><span class="o">=</span><span class="s2">"2018-04-16T20:01:14Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>info<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Starting DNS over HTTPS proxy server"</span><span class="w"> </span><span class="nv">addr</span><span class="o">=</span><span class="s2">"dns://0.0.0.0:5054"</span>
cloudflared<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="nv">time</span><span class="o">=</span><span class="s2">"2018-04-16T20:01:14Z"</span><span class="w"> </span><span class="nv">level</span><span class="o">=</span>info<span class="w"> </span><span class="nv">msg</span><span class="o">=</span><span class="s2">"Starting metrics server"</span><span class="w"> </span><span class="nv">addr</span><span class="o">=</span><span class="s2">"127.0.0.1:35973"</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="n">docker</span><span class="o">-</span><span class="n">compose</span><span class="w"> </span><span class="n">logs</span><span class="w"> </span><span class="nf">pi</span><span class="o">-</span><span class="n">hole</span>
<span class="n">Attaching</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="nf">pi</span><span class="o">-</span><span class="n">hole</span>
<span class="p">...</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">[</span><span class="n">services.d</span><span class="o">]</span><span class="w"> </span><span class="n">starting</span><span class="w"> </span><span class="n">services</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">lighttpd</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">dnsmasq</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">crond</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">pihole</span><span class="o">-</span><span class="n">FTL</span><span class="w"> </span><span class="p">(</span><span class="k">no</span><span class="o">-</span><span class="n">daemon</span><span class="p">)</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">[</span><span class="n">services.d</span><span class="o">]</span><span class="w"> </span><span class="n">done</span><span class="p">.</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nl">dnsmasq</span><span class="p">:</span><span class="w"> </span><span class="n">started</span><span class="p">,</span><span class="w"> </span><span class="n">version</span><span class="w"> </span><span class="mf">2.76</span><span class="w"> </span><span class="n">cachesize</span><span class="w"> </span><span class="mi">10000</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nl">dnsmasq</span><span class="p">:</span><span class="w"> </span><span class="n">compile</span><span class="w"> </span><span class="nc">time</span><span class="w"> </span><span class="nl">options</span><span class="p">:</span><span class="w"> </span><span class="n">IPv6</span><span class="w"> </span><span class="n">GNU</span><span class="o">-</span><span class="n">getopt</span><span class="w"> </span><span class="n">DBus</span><span class="w"> </span><span class="n">i18n</span><span class="w"> </span><span class="n">IDN</span><span class="w"> </span><span class="n">DHCP</span><span class="w"> </span><span class="n">DHCPv6</span><span class="w"> </span><span class="k">no</span><span class="o">-</span><span class="n">Lua</span><span class="w"> </span><span class="n">TFTP</span><span class="w"> </span><span class="n">conntrack</span><span class="w"> </span><span class="n">ipset</span><span class="w"> </span><span class="n">auth</span><span class="w"> </span><span class="n">DNSSEC</span><span class="w"> </span><span class="n">loop</span><span class="o">-</span><span class="n">detect</span><span class="w"> </span><span class="n">inotify</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nl">dnsmasq</span><span class="p">:</span><span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">nameserver</span><span class="w"> </span><span class="mf">10.0.0.2</span><span class="n">#5054</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nl">dnsmasq</span><span class="p">:</span><span class="w"> </span><span class="k">read</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">hosts</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="n">addresses</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nl">dnsmasq</span><span class="p">:</span><span class="w"> </span><span class="k">read</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">pihole</span><span class="o">/</span><span class="k">local</span><span class="p">.</span><span class="n">list</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">addresses</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nl">dnsmasq</span><span class="p">:</span><span class="w"> </span><span class="n">failed</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">load</span><span class="w"> </span><span class="k">names</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">pihole</span><span class="o">/</span><span class="n">black</span><span class="p">.</span><span class="nl">list</span><span class="p">:</span><span class="w"> </span><span class="k">No</span><span class="w"> </span><span class="n">such</span><span class="w"> </span><span class="k">file</span><span class="w"> </span><span class="ow">or</span><span class="w"> </span><span class="n">directory</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nl">dnsmasq</span><span class="p">:</span><span class="w"> </span><span class="k">read</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">pihole</span><span class="o">/</span><span class="n">gravity</span><span class="p">.</span><span class="n">list</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">121065</span><span class="w"> </span><span class="n">addresses</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nl">dnsmasq</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mf">127.0.0.1</span><span class="o">/</span><span class="mi">48521</span><span class="w"> </span><span class="n">query</span><span class="o">[</span><span class="n">A</span><span class="o">]</span><span class="w"> </span><span class="nf">pi</span><span class="p">.</span><span class="n">hole</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="mf">127.0.0.1</span>
<span class="nf">pi</span><span class="o">-</span><span class="n">hole</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nl">dnsmasq</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mf">127.0.0.1</span><span class="o">/</span><span class="mi">48521</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">pihole</span><span class="o">/</span><span class="k">local</span><span class="p">.</span><span class="n">list</span><span class="w"> </span><span class="nf">pi</span><span class="p">.</span><span class="n">hole</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="mf">10.0.0.3</span>
</code></pre></div>
<p>you should be able to query the containerized pi-hole DNS service from it's host or from within your netwerk using dig:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>dig<span class="w"> </span>@localhost<span class="w"> </span>-p<span class="w"> </span><span class="m">53</span><span class="w"> </span>visibilityspots.org
<span class="o">(</span>$<span class="w"> </span>dig<span class="w"> </span>@IP-ADDRESS-OF-DOCKER-NODE<span class="w"> </span>-p<span class="w"> </span><span class="m">53</span><span class="w"> </span>visibilityspots.org<span class="o">)</span>
<span class="p">;</span><span class="w"> </span><<>><span class="w"> </span>DiG<span class="w"> </span><span class="m">9</span>.12.1<span class="w"> </span><<>><span class="w"> </span>@localhost<span class="w"> </span>-p<span class="w"> </span><span class="m">53</span><span class="w"> </span>visibilityspots.org
<span class="p">;</span><span class="w"> </span><span class="o">(</span><span class="m">2</span><span class="w"> </span>servers<span class="w"> </span>found<span class="o">)</span>
<span class="p">;;</span><span class="w"> </span>global<span class="w"> </span>options:<span class="w"> </span>+cmd
<span class="p">;;</span><span class="w"> </span>Got<span class="w"> </span>answer:
<span class="p">;;</span><span class="w"> </span>->>HEADER<span class="s"><<- opco</span>de:<span class="w"> </span>QUERY,<span class="w"> </span>status:<span class="w"> </span>NOERROR,<span class="w"> </span>id:<span class="w"> </span><span class="m">51155</span>
<span class="p">;;</span><span class="w"> </span>flags:<span class="w"> </span>qr<span class="w"> </span>rd<span class="w"> </span>ra<span class="p">;</span><span class="w"> </span>QUERY:<span class="w"> </span><span class="m">1</span>,<span class="w"> </span>ANSWER:<span class="w"> </span><span class="m">8</span>,<span class="w"> </span>AUTHORITY:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span>ADDITIONAL:<span class="w"> </span><span class="m">1</span>
<span class="p">;;</span><span class="w"> </span>OPT<span class="w"> </span>PSEUDOSECTION:
<span class="p">;</span><span class="w"> </span>EDNS:<span class="w"> </span>version:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span>flags:<span class="p">;</span><span class="w"> </span>udp:<span class="w"> </span><span class="m">1536</span>
<span class="p">;;</span><span class="w"> </span>QUESTION<span class="w"> </span>SECTION:
<span class="p">;</span>visibilityspots.org.<span class="w"> </span>IN<span class="w"> </span>A
<span class="p">;;</span><span class="w"> </span>ANSWER<span class="w"> </span>SECTION:
visibilityspots.org.<span class="w"> </span><span class="m">37</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">54</span>.230.9.72
visibilityspots.org.<span class="w"> </span><span class="m">37</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">54</span>.230.9.109
visibilityspots.org.<span class="w"> </span><span class="m">37</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">54</span>.230.9.119
visibilityspots.org.<span class="w"> </span><span class="m">37</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">54</span>.230.9.143
visibilityspots.org.<span class="w"> </span><span class="m">37</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">54</span>.230.9.148
visibilityspots.org.<span class="w"> </span><span class="m">37</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">54</span>.230.9.182
visibilityspots.org.<span class="w"> </span><span class="m">37</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">54</span>.230.9.188
visibilityspots.org.<span class="w"> </span><span class="m">37</span><span class="w"> </span>IN<span class="w"> </span>A<span class="w"> </span><span class="m">54</span>.230.9.203
<span class="p">;;</span><span class="w"> </span>Query<span class="w"> </span>time:<span class="w"> </span><span class="m">223</span><span class="w"> </span>msec
<span class="p">;;</span><span class="w"> </span>SERVER:<span class="w"> </span>::1#53<span class="o">(</span>::1<span class="o">)</span>
<span class="p">;;</span><span class="w"> </span>WHEN:<span class="w"> </span>Mon<span class="w"> </span>Apr<span class="w"> </span><span class="m">16</span><span class="w"> </span><span class="m">22</span>:05:37<span class="w"> </span>CEST<span class="w"> </span><span class="m">2018</span>
<span class="p">;;</span><span class="w"> </span>MSG<span class="w"> </span>SIZE<span class="w"> </span>rcvd:<span class="w"> </span><span class="m">328</span>
</code></pre></div>
<p>Obviously I wanted to see myself that when sniffing the network the DNS requests aren't readable so I used tcp dump to prove myself the data was sent through HTTPS</p>
<div class="highlight"><pre><span></span><code>pi-hole# tcpdump -i eth0 udp port 53
22:39:30.837594 IP 192.168.0.3.35765 > piholeContainerID.domain: 36972+ [1au] A? visibilityspots.com. (60)
22:39:31.009345 IP piholeContainerID.domain > 192.168.0.3.35765: 36972 8/0/1 A 54.230.228.38, A 54.230.228.42, A 54.230.228.54, A 54.230.228.68, A 54.230.228.69, A 54.230.228.84, A 54.230.228.92, A 54.230.228.104 (328)
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="nx">cloudflared</span><span class="err">#</span><span class="w"> </span><span class="nx">tcpdump</span><span class="w"> </span><span class="o">-</span><span class="nx">i</span><span class="w"> </span><span class="nx">eth0</span><span class="w"> </span><span class="nx">udp</span><span class="w"> </span><span class="nx">port</span><span class="w"> </span><span class="mi">5054</span>
<span class="nx">tcpdump</span><span class="p">:</span><span class="w"> </span><span class="nx">verbose</span><span class="w"> </span><span class="nx">output</span><span class="w"> </span><span class="nx">suppressed</span><span class="p">,</span><span class="w"> </span><span class="nx">use</span><span class="w"> </span><span class="o">-</span><span class="nx">v</span><span class="w"> </span><span class="k">or</span><span class="w"> </span><span class="o">-</span><span class="nx">vv</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">full</span><span class="w"> </span><span class="nx">protocol</span><span class="w"> </span><span class="nx">decode</span>
<span class="nx">listening</span><span class="w"> </span><span class="nx">on</span><span class="w"> </span><span class="nx">eth0</span><span class="p">,</span><span class="w"> </span><span class="nx">link</span><span class="o">-</span><span class="k">type</span><span class="w"> </span><span class="nx">EN10MB</span><span class="w"> </span><span class="p">(</span><span class="nx">Ethernet</span><span class="p">),</span><span class="w"> </span><span class="nx">capture</span><span class="w"> </span><span class="nx">size</span><span class="w"> </span><span class="mi">262144</span><span class="w"> </span><span class="nx">bytes</span>
<span class="mi">20</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="m m-Double">29.029132</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">piholeContainerID</span><span class="m m-Double">.59189</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">cloudflaredContainerID</span><span class="m m-Double">.5054</span><span class="p">:</span><span class="w"> </span><span class="nx">UDP</span><span class="p">,</span><span class="w"> </span><span class="nx">length</span><span class="w"> </span><span class="mi">40</span>
<span class="mi">20</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="m m-Double">29.069864</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">cloudflaredContainerID</span><span class="m m-Double">.5054</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">piholeContainerID</span><span class="m m-Double">.59189</span><span class="p">:</span><span class="w"> </span><span class="nx">UDP</span><span class="p">,</span><span class="w"> </span><span class="nx">length</span><span class="w"> </span><span class="mi">116</span>
<span class="mi">20</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="m m-Double">30.838803</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">piholeContainerID</span><span class="m m-Double">.28892</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">cloudflaredContainerID</span><span class="m m-Double">.5054</span><span class="p">:</span><span class="w"> </span><span class="nx">UDP</span><span class="p">,</span><span class="w"> </span><span class="nx">length</span><span class="w"> </span><span class="mi">60</span>
<span class="mi">20</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="m m-Double">31.003756</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">cloudflaredContainerID</span><span class="m m-Double">.5054</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">piholeContainerID</span><span class="m m-Double">.28892</span><span class="p">:</span><span class="w"> </span><span class="nx">UDP</span><span class="p">,</span><span class="w"> </span><span class="nx">length</span><span class="w"> </span><span class="mi">328</span>
<span class="mi">20</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="m m-Double">31.352487</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">piholeContainerID</span><span class="m m-Double">.50291</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">cloudflaredContainerID</span><span class="m m-Double">.5054</span><span class="p">:</span><span class="w"> </span><span class="nx">UDP</span><span class="p">,</span><span class="w"> </span><span class="nx">length</span><span class="w"> </span><span class="mi">31</span>
<span class="mi">20</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="m m-Double">31.364073</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">piholeContainerID</span><span class="m m-Double">.16365</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">cloudflaredContainerID</span><span class="m m-Double">.5054</span><span class="p">:</span><span class="w"> </span><span class="nx">UDP</span><span class="p">,</span><span class="w"> </span><span class="nx">length</span><span class="w"> </span><span class="mi">31</span>
<span class="mi">20</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="m m-Double">31.411227</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">cloudflaredContainerID</span><span class="m m-Double">.5054</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">piholeContainerID</span><span class="m m-Double">.50291</span><span class="p">:</span><span class="w"> </span><span class="nx">UDP</span><span class="p">,</span><span class="w"> </span><span class="nx">length</span><span class="w"> </span><span class="mi">156</span>
<span class="mi">20</span><span class="p">:</span><span class="mi">39</span><span class="p">:</span><span class="m m-Double">31.432364</span><span class="w"> </span><span class="nx">IP</span><span class="w"> </span><span class="nx">cloudflaredContainerID</span><span class="m m-Double">.5054</span><span class="w"> </span><span class="p">></span><span class="w"> </span><span class="nx">piholeContainerID</span><span class="m m-Double">.16365</span><span class="p">:</span><span class="w"> </span><span class="nx">UDP</span><span class="p">,</span><span class="w"> </span><span class="nx">length</span><span class="w"> </span><span class="mi">218</span>
</code></pre></div>
<p>So by now you can configure this new DNS service on your router or dhcp daemon within your local network.</p>
<p>Since the pi isn't running for a very long time I have no clue if it can cope with the load on our network but I'll keep you posted ;)</p>NRPE troubleshooting2018-02-15T21:00:00+01:002018-02-15T00:00:00+01:00Jantag:visibilityspots.org,2018-02-15:/nrpe-troubleshooting.html<p>When refactoring a <a href="https://github.com/visibilityspots/icinga-scripts/blob/master/check_memory">check_memory</a> I wrote a few years ago I bumped into the feared</p>
<div class="highlight"><pre><span></span><code><span class="n">NRPE</span><span class="o">:</span><span class="w"> </span><span class="n">Unable</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">read</span><span class="w"> </span><span class="n">output</span>
</code></pre></div>
<p>error message on our nagios instance.</p>
<p>When looking for a solution I went through most possible debug steps I could think of and which are nicely described by nagios <a href="https://support.nagios.com/kb/article/nrpe-nrpe-unable-to-read-output-620.html">support</a> but didn't found any solution.</p>
<p>I almost grabbed to some anti depressants when I thought of the thing I always forget about.</p>
<p><strong>SELINUX</strong></p>
<p>When crawling through the audit log it became clear I forgot to configure the proper selinux context type for the new script.</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span><span class="p">=</span><span class="nx">PATH</span><span class="w"> </span><span class="nx">msg</span><span class="p">=</span><span class="nx">audit</span><span class="p">(</span><span class="m m-Double">1518702310 …</span></code></pre></div><p>When refactoring a <a href="https://github.com/visibilityspots/icinga-scripts/blob/master/check_memory">check_memory</a> I wrote a few years ago I bumped into the feared</p>
<div class="highlight"><pre><span></span><code><span class="n">NRPE</span><span class="o">:</span><span class="w"> </span><span class="n">Unable</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">read</span><span class="w"> </span><span class="n">output</span>
</code></pre></div>
<p>error message on our nagios instance.</p>
<p>When looking for a solution I went through most possible debug steps I could think of and which are nicely described by nagios <a href="https://support.nagios.com/kb/article/nrpe-nrpe-unable-to-read-output-620.html">support</a> but didn't found any solution.</p>
<p>I almost grabbed to some anti depressants when I thought of the thing I always forget about.</p>
<p><strong>SELINUX</strong></p>
<p>When crawling through the audit log it became clear I forgot to configure the proper selinux context type for the new script.</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span><span class="p">=</span><span class="nx">PATH</span><span class="w"> </span><span class="nx">msg</span><span class="p">=</span><span class="nx">audit</span><span class="p">(</span><span class="m m-Double">1518702310.763</span><span class="p">:</span><span class="mi">296695</span><span class="p">):</span><span class="w"> </span><span class="nx">item</span><span class="p">=</span><span class="mi">0</span><span class="w"> </span><span class="nx">name</span><span class="p">=</span><span class="s">"/usr/lib64/nagios/plugins/check_mem"</span><span class="w"> </span><span class="nx">inode</span><span class="p">=</span><span class="mi">1126947</span><span class="w"> </span><span class="nx">dev</span><span class="p">=</span><span class="nx">fd</span><span class="p">:</span><span class="mi">00</span><span class="w"> </span><span class="nx">mode</span><span class="p">=</span><span class="mi">0100755</span><span class="w"> </span><span class="nx">ouid</span><span class="p">=</span><span class="mi">0</span><span class="w"> </span><span class="nx">ogid</span><span class="p">=</span><span class="mi">0</span><span class="w"> </span><span class="nx">rdev</span><span class="p">=</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="w"> </span><span class="nx">obj</span><span class="p">=</span><span class="nx">unconfined_u</span><span class="p">:</span><span class="nx">object_r</span><span class="p">:</span><span class="nx">admin_home_t</span><span class="p">:</span><span class="nx">s0</span><span class="w"> </span><span class="nx">objtype</span><span class="p">=</span><span class="nx">NORMAL</span>
<span class="k">type</span><span class="p">=</span><span class="nx">PROCTITLE</span><span class="w"> </span><span class="nx">msg</span><span class="p">=</span><span class="nx">audit</span><span class="p">(</span><span class="m m-Double">1518702310.763</span><span class="p">:</span><span class="mi">296695</span><span class="p">):</span><span class="w"> </span><span class="nx">proctitle</span><span class="p">=</span><span class="mi">7368002</span><span class="nx">D63002F7573722F6C696236342F6E6167696F732F706C7567696E732F636865636B5F6D656D2E7368</span>
<span class="k">type</span><span class="p">=</span><span class="nx">AVC</span><span class="w"> </span><span class="nx">msg</span><span class="p">=</span><span class="nx">audit</span><span class="p">(</span><span class="m m-Double">1518702310.763</span><span class="p">:</span><span class="mi">296696</span><span class="p">):</span><span class="w"> </span><span class="nx">avc</span><span class="p">:</span><span class="w"> </span><span class="nx">denied</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">getattr</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">pid</span><span class="p">=</span><span class="mi">21213</span><span class="w"> </span><span class="nx">comm</span><span class="p">=</span><span class="s">"sh"</span><span class="w"> </span><span class="nx">path</span><span class="p">=</span><span class="s">"/usr/lib64/nagios/plugins/check_mem"</span><span class="w"> </span><span class="nx">dev</span><span class="p">=</span><span class="s">"dm-0"</span><span class="w"> </span><span class="nx">ino</span><span class="p">=</span><span class="mi">1126947</span><span class="w"> </span><span class="nx">scontext</span><span class="p">=</span><span class="nx">system_u</span><span class="p">:</span><span class="nx">system_r</span><span class="p">:</span><span class="nx">nrpe_t</span><span class="p">:</span><span class="nx">s0</span><span class="w"> </span><span class="nx">tcontext</span><span class="p">=</span><span class="nx">unconfined_u</span><span class="p">:</span><span class="nx">object_r</span><span class="p">:</span><span class="nx">admin_home_t</span><span class="p">:</span><span class="nx">s0</span><span class="w"> </span><span class="nx">tclass</span><span class="p">=</span><span class="nx">file</span>
<span class="k">type</span><span class="p">=</span><span class="nx">SYSCALL</span><span class="w"> </span><span class="nx">msg</span><span class="p">=</span><span class="nx">audit</span><span class="p">(</span><span class="m m-Double">1518702310.763</span><span class="p">:</span><span class="mi">296696</span><span class="p">):</span><span class="w"> </span><span class="nx">arch</span><span class="p">=</span><span class="nx">c000003e</span><span class="w"> </span><span class="nx">syscall</span><span class="p">=</span><span class="mi">4</span><span class="w"> </span><span class="nx">success</span><span class="p">=</span><span class="nx">no</span><span class="w"> </span><span class="nx">exit</span><span class="p">=</span><span class="o">-</span><span class="mi">13</span><span class="w"> </span><span class="nx">a0</span><span class="p">=</span><span class="mi">268</span><span class="nx">ea10</span><span class="w"> </span><span class="nx">a1</span><span class="p">=</span><span class="mi">7</span><span class="nx">ffc5f708730</span><span class="w"> </span><span class="nx">a2</span><span class="p">=</span><span class="mi">7</span><span class="nx">ffc5f708730</span><span class="w"> </span><span class="nx">a3</span><span class="p">=</span><span class="mi">7</span><span class="nx">ffc5f708260</span><span class="w"> </span><span class="nx">items</span><span class="p">=</span><span class="mi">1</span><span class="w"> </span><span class="nx">ppid</span><span class="p">=</span><span class="mi">21212</span><span class="w"> </span><span class="nx">pid</span><span class="p">=</span><span class="mi">21213</span><span class="w"> </span><span class="nx">auid</span><span class="p">=</span><span class="mi">4294967295</span><span class="w"> </span><span class="nx">uid</span><span class="p">=</span><span class="mi">997</span><span class="w"> </span><span class="nx">gid</span><span class="p">=</span><span class="mi">994</span><span class="w"> </span><span class="nx">euid</span><span class="p">=</span><span class="mi">997</span><span class="w"> </span><span class="nx">suid</span><span class="p">=</span><span class="mi">997</span><span class="w"> </span><span class="nx">fsuid</span><span class="p">=</span><span class="mi">997</span><span class="w"> </span><span class="nx">egid</span><span class="p">=</span><span class="mi">994</span><span class="w"> </span><span class="nx">sgid</span><span class="p">=</span><span class="mi">994</span><span class="w"> </span><span class="nx">fsgid</span><span class="p">=</span><span class="mi">994</span><span class="w"> </span><span class="nx">tty</span><span class="p">=(</span><span class="nx">none</span><span class="p">)</span><span class="w"> </span><span class="nx">ses</span><span class="p">=</span><span class="mi">4294967295</span><span class="w"> </span><span class="nx">comm</span><span class="p">=</span><span class="s">"sh"</span><span class="w"> </span><span class="nx">exe</span><span class="p">=</span><span class="s">"/usr/bin/bash"</span><span class="w"> </span><span class="nx">subj</span><span class="p">=</span><span class="nx">system_u</span><span class="p">:</span><span class="nx">system_r</span><span class="p">:</span><span class="nx">nrpe_t</span><span class="p">:</span><span class="nx">s0</span><span class="w"> </span><span class="nx">key</span><span class="p">=(</span><span class="nx">null</span><span class="p">)</span>
</code></pre></div>
<p>By refreshing my memory about selinux again following this gentoo <a href="https://wiki.gentoo.org/wiki/SELinux/Tutorials/Where_to_find_SELinux_permission_denial_details">tutorial</a> I could quickly fix the issue by configuring the proper context type;</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="nx">semanage</span><span class="w"> </span><span class="nx">fcontext</span><span class="w"> </span><span class="o">-</span><span class="nx">a</span><span class="w"> </span><span class="o">--</span><span class="k">type</span><span class="w"> </span><span class="nx">nagios_unconfined_plugin_exec_t</span><span class="w"> </span><span class="o">/</span><span class="nx">usr</span><span class="o">/</span><span class="nx">lib64</span><span class="o">/</span><span class="nx">nagios</span><span class="o">/</span><span class="nx">plugins</span><span class="o">/</span><span class="nx">check_mem</span>
<span class="err">#</span><span class="w"> </span><span class="nx">restorecon</span><span class="w"> </span><span class="o">/</span><span class="nx">usr</span><span class="o">/</span><span class="nx">lib64</span><span class="o">/</span><span class="nx">nagios</span><span class="o">/</span><span class="nx">plugins</span><span class="o">/</span><span class="nx">check_mem</span>
</code></pre></div>
<p>and running the check_nrpe tool from the nagios instance finally worked again as I expected it to be:</p>
<div class="highlight"><pre><span></span><code><span class="gh">#</span> /usr/lib64/nagios/plugins/check_nrpe -H host -c check_mem
OK: 63% of memory used, 48% is available
</code></pre></div>Test ansible playbooks with docker2017-11-09T21:00:00+01:002017-11-09T00:00:00+01:00Jantag:visibilityspots.org,2017-11-09:/test-ansible-playbooks.html<p>recently I started working at a new project where the infra is maintained by ansible. When been asked to write some functionality in a playbook I missed my <a href="https://github.com/visibilityspots/vagrant-puppet">vagrant puppet</a> setup where I could easily test my puppet code on my local machine.</p>
<p>Due to my previous project I felt like maybe I could use docker for this purpose on the ansible part. So I looked a bit around and stumbled on the <a href="https://github.com/William-Yeh/docker-ansible">docker-ansible github repository</a> of William Yeh. He already did a great job by creating a docker container with ansible preinstalled for a lot of linux distributions.</p>
<p>I …</p><p>recently I started working at a new project where the infra is maintained by ansible. When been asked to write some functionality in a playbook I missed my <a href="https://github.com/visibilityspots/vagrant-puppet">vagrant puppet</a> setup where I could easily test my puppet code on my local machine.</p>
<p>Due to my previous project I felt like maybe I could use docker for this purpose on the ansible part. So I looked a bit around and stumbled on the <a href="https://github.com/William-Yeh/docker-ansible">docker-ansible github repository</a> of William Yeh. He already did a great job by creating a docker container with ansible preinstalled for a lot of linux distributions.</p>
<p>I only figured the way he describes wasn't really what I am looking for. By using docker build the test of the playbook creates a docker image every time. I was looking for a solution a docker container will be brought up, the ansible playbook being tested, showing those results and bringing the container back down without too much hassle to set this environment up and running.</p>
<p>So I digged around a bit further in his images and came up with the following command</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>-ti<span class="w"> </span>--rm<span class="w"> </span>-v<span class="w"> </span><span class="s2">"</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span><span class="s2">"</span>:/tmp<span class="w"> </span>--workdir<span class="o">=</span><span class="s2">"/tmp"</span><span class="w"> </span>williamyeh/ansible:centos7-onbuild<span class="w"> </span>ansible-playbook-wrapper
</code></pre></div>
<p>This will mount the current directory where a file named playbook.yml is placed to the /tmp directory of the docker container. By changing the container's workdir to this /tmp directory the ansible-playbook-wrapper he wrote will be executed using the mounted playbook.yml and spawning the results.</p>
<div class="highlight"><pre><span></span><code><span class="n">$</span><span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="o">-</span><span class="n">ti</span><span class="w"> </span><span class="o">--</span><span class="n">rm</span><span class="w"> </span><span class="o">-</span><span class="n">v</span><span class="w"> </span><span class="s">"$(pwd)"</span><span class="o">:/</span><span class="n">tmp</span><span class="w"> </span><span class="o">--</span><span class="n">workdir</span><span class="o">=</span><span class="s">"/tmp"</span><span class="w"> </span><span class="n">williamyeh</span><span class="o">/</span><span class="n">ansible</span><span class="o">:</span><span class="n">centos7</span><span class="o">-</span><span class="n">onbuild</span><span class="w"> </span><span class="n">ansible</span><span class="o">-</span><span class="n">playbook</span><span class="o">-</span><span class="n">wrapper</span>
<span class="n">PLAY</span><span class="w"> </span><span class="p">[</span><span class="n">Remove</span><span class="w"> </span><span class="n">upstream</span><span class="w"> </span><span class="n">repositories</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="n">being</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="n">offline</span><span class="w"> </span><span class="n">instance</span><span class="p">]</span><span class="w"> </span><span class="o">**************************************************************************************************************************************************************************************************************</span>
<span class="n">TASK</span><span class="w"> </span><span class="p">[</span><span class="n">Gathering</span><span class="w"> </span><span class="n">Facts</span><span class="p">]</span><span class="w"> </span><span class="o">**********************************************************************************************************************************************************************************************************************************************************</span>
<span class="nl">ok</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span>
<span class="n">TASK</span><span class="w"> </span><span class="p">[</span><span class="n">Check</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">supposed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">offline</span><span class="p">]</span><span class="w"> </span><span class="o">*********************************************************************************************************************************************************************************************************************************</span>
<span class="nl">ok</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span>
<span class="n">TASK</span><span class="w"> </span><span class="p">[</span><span class="n">find</span><span class="p">]</span><span class="w"> </span><span class="o">*********************************************************************************************************************************************************************************************************************************************************************</span>
<span class="nl">ok</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span>
<span class="n">TASK</span><span class="w"> </span><span class="p">[</span><span class="n">file</span><span class="p">]</span><span class="w"> </span><span class="o">*********************************************************************************************************************************************************************************************************************************************************************</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">560</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1664</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Base</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">60</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">561</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1309</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">CR</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">60</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">562</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">649</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Debuginfo</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">60</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">563</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">630</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Media</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">60</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">564</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1331</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Sources</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">60</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">565</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">3830</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Vault</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">60</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">skipping</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">566</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">314</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">fasttrack</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">60</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="n">PLAY</span><span class="w"> </span><span class="n">RECAP</span><span class="w"> </span><span class="o">**********************************************************************************************************************************************************************************************************************************************************************</span>
<span class="nl">localhost</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ok</span><span class="o">=</span><span class="mi">4</span><span class="w"> </span><span class="n">changed</span><span class="o">=</span><span class="mi">1</span><span class="w"> </span><span class="n">unreachable</span><span class="o">=</span><span class="mi">0</span><span class="w"> </span><span class="n">failed</span><span class="o">=</span><span class="mi">0</span>
</code></pre></div>
<p>the playbook will check for a certain file and depending on that file's existence it will remove all repositories except from a predefined list.</p>
<div class="highlight"><pre><span></span><code><span class="o">---</span>
<span class="o">-</span><span class="w"> </span><span class="n">name</span><span class="p">:</span><span class="w"> </span><span class="n">Remove</span><span class="w"> </span><span class="n">upstream</span><span class="w"> </span><span class="n">repositories</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="n">being</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="n">offline</span><span class="w"> </span><span class="n">instance</span>
<span class="w"> </span><span class="n">hosts</span><span class="p">:</span><span class="w"> </span><span class="n">localhost</span>
<span class="w"> </span><span class="n">become_user</span><span class="p">:</span><span class="w"> </span><span class="n">root</span>
<span class="w"> </span><span class="n">vars</span><span class="p">:</span>
<span class="w"> </span><span class="n">repos_to_keep</span><span class="p">:</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="o">.</span><span class="n">repos</span><span class="o">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">fasttrack</span><span class="o">.</span><span class="n">repo</span>
<span class="w"> </span><span class="n">tasks</span><span class="p">:</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">name</span><span class="p">:</span><span class="w"> </span><span class="n">Check</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">supposed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">offline</span>
<span class="w"> </span><span class="n">stat</span><span class="p">:</span>
<span class="w"> </span><span class="n">path</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">opt</span><span class="o">/</span><span class="n">offline</span>
<span class="w"> </span><span class="n">register</span><span class="p">:</span><span class="w"> </span><span class="n">offline</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">find</span><span class="p">:</span>
<span class="w"> </span><span class="n">paths</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="o">.</span><span class="n">repos</span><span class="o">.</span><span class="n">d</span><span class="o">/</span>
<span class="w"> </span><span class="n">patterns</span><span class="p">:</span><span class="w"> </span><span class="s2">"*.repo"</span>
<span class="w"> </span><span class="n">register</span><span class="p">:</span><span class="w"> </span><span class="n">repos</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">debug</span><span class="p">:</span>
<span class="w"> </span><span class="k">var</span><span class="p">:</span><span class="w"> </span><span class="n">repos</span><span class="o">.</span><span class="n">files</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">file</span><span class="p">:</span>
<span class="w"> </span><span class="n">path</span><span class="p">:</span><span class="w"> </span><span class="s2">"{{ item.path }}"</span>
<span class="w"> </span><span class="n">state</span><span class="p">:</span><span class="w"> </span><span class="n">absent</span>
<span class="w"> </span><span class="n">with_items</span><span class="p">:</span><span class="w"> </span><span class="s2">"{{ repos.files }}"</span>
<span class="w"> </span><span class="n">when</span><span class="p">:</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">item</span><span class="o">.</span><span class="n">path</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">repos_to_keep</span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">offline</span><span class="o">.</span><span class="n">stat</span><span class="o">.</span><span class="n">exists</span>
</code></pre></div>
<p>during development you want to keep the container online to troubleshoot. This can be done by running the container and instead of executing the ansible-playbook-wrapper just launching bash.</p>
<div class="highlight"><pre><span></span><code><span class="n">$</span><span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="o">-</span><span class="n">ti</span><span class="w"> </span><span class="o">--</span><span class="n">rm</span><span class="w"> </span><span class="o">-</span><span class="n">v</span><span class="w"> </span><span class="s">"$(pwd)"</span><span class="o">:/</span><span class="n">tmp</span><span class="w"> </span><span class="o">--</span><span class="n">workdir</span><span class="o">=</span><span class="s">"/tmp"</span><span class="w"> </span><span class="n">williamyeh</span><span class="o">/</span><span class="n">ansible</span><span class="o">:</span><span class="n">centos7</span><span class="o">-</span><span class="n">onbuild</span><span class="w"> </span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">bash</span>
<span class="p">[</span><span class="n">root</span><span class="mi">@5</span><span class="n">cd5cfe2d7cf</span><span class="w"> </span><span class="n">tmp</span><span class="p">]</span><span class="err">#</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span>
<span class="n">CentOS</span><span class="o">-</span><span class="n">Base</span><span class="p">.</span><span class="n">repo</span><span class="w"> </span><span class="n">CentOS</span><span class="o">-</span><span class="n">CR</span><span class="p">.</span><span class="n">repo</span><span class="w"> </span><span class="n">CentOS</span><span class="o">-</span><span class="n">Debuginfo</span><span class="p">.</span><span class="n">repo</span><span class="w"> </span><span class="n">CentOS</span><span class="o">-</span><span class="n">fasttrack</span><span class="p">.</span><span class="n">repo</span><span class="w"> </span><span class="n">CentOS</span><span class="o">-</span><span class="n">Media</span><span class="p">.</span><span class="n">repo</span><span class="w"> </span><span class="n">CentOS</span><span class="o">-</span><span class="n">Sources</span><span class="p">.</span><span class="n">repo</span><span class="w"> </span><span class="n">CentOS</span><span class="o">-</span><span class="n">Vault</span><span class="p">.</span><span class="n">repo</span>
<span class="p">[</span><span class="n">root</span><span class="mi">@5</span><span class="n">cd5cfe2d7cf</span><span class="w"> </span><span class="n">tmp</span><span class="p">]</span><span class="err">#</span><span class="w"> </span><span class="n">ansible</span><span class="o">-</span><span class="n">playbook</span><span class="o">-</span><span class="n">wrapper</span>
<span class="n">PLAY</span><span class="w"> </span><span class="p">[</span><span class="n">Remove</span><span class="w"> </span><span class="n">upstream</span><span class="w"> </span><span class="n">repositories</span><span class="w"> </span><span class="n">when</span><span class="w"> </span><span class="n">being</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="n">offline</span><span class="w"> </span><span class="n">instance</span><span class="p">]</span><span class="w"> </span><span class="o">**************************************************************************************************************************************************************************************************************</span>
<span class="n">TASK</span><span class="w"> </span><span class="p">[</span><span class="n">Gathering</span><span class="w"> </span><span class="n">Facts</span><span class="p">]</span><span class="w"> </span><span class="o">**********************************************************************************************************************************************************************************************************************************************************</span>
<span class="nl">ok</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span>
<span class="n">TASK</span><span class="w"> </span><span class="p">[</span><span class="n">Check</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">system</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">supposed</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">offline</span><span class="p">]</span><span class="w"> </span><span class="o">*********************************************************************************************************************************************************************************************************************************</span>
<span class="nl">ok</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span>
<span class="n">TASK</span><span class="w"> </span><span class="p">[</span><span class="n">find</span><span class="p">]</span><span class="w"> </span><span class="o">*********************************************************************************************************************************************************************************************************************************************************************</span>
<span class="nl">ok</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span>
<span class="n">TASK</span><span class="w"> </span><span class="p">[</span><span class="n">file</span><span class="p">]</span><span class="w"> </span><span class="o">*********************************************************************************************************************************************************************************************************************************************************************</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">560</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1664</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Base</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">59</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">561</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1309</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">CR</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">59</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">562</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">649</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Debuginfo</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">59</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">563</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">630</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Media</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">59</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">564</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1331</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Sources</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">59</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">changed</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">565</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">3830</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">Vault</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">59</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="nl">skipping</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="n">localhost</span><span class="p">]</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="p">{</span><span class="n">u</span><span class="err">'</span><span class="n">uid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">woth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mtime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">inode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">566</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isgid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">size</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">314</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">roth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isuid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isreg</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">gid</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ischr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xoth</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">nlink</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">issock</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">rgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">True</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">path</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="n">CentOS</span><span class="o">-</span><span class="n">fasttrack</span><span class="p">.</span><span class="n">repo</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xusr</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">atime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1504108387.0</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isdir</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">ctime</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mf">1510218719.8578098</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">wgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">xgrp</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">dev</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="mi">59</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isblk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">isfifo</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">mode</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="mo">0644</span><span class="err">'</span><span class="p">,</span><span class="w"> </span><span class="n">u</span><span class="err">'</span><span class="n">islnk</span><span class="err">'</span><span class="o">:</span><span class="w"> </span><span class="n">False</span><span class="p">})</span>
<span class="n">PLAY</span><span class="w"> </span><span class="n">RECAP</span><span class="w"> </span><span class="o">**********************************************************************************************************************************************************************************************************************************************************************</span>
<span class="nl">localhost</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="n">ok</span><span class="o">=</span><span class="mi">4</span><span class="w"> </span><span class="n">changed</span><span class="o">=</span><span class="mi">1</span><span class="w"> </span><span class="n">unreachable</span><span class="o">=</span><span class="mi">0</span><span class="w"> </span><span class="n">failed</span><span class="o">=</span><span class="mi">0</span>
<span class="p">[</span><span class="n">root</span><span class="mi">@5</span><span class="n">cd5cfe2d7cf</span><span class="w"> </span><span class="n">tmp</span><span class="p">]</span><span class="err">#</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">yum</span><span class="p">.</span><span class="n">repos</span><span class="p">.</span><span class="n">d</span><span class="o">/</span>
<span class="n">CentOS</span><span class="o">-</span><span class="n">fasttrack</span><span class="p">.</span><span class="n">repo</span>
<span class="p">[</span><span class="n">root</span><span class="mi">@5</span><span class="n">cd5cfe2d7cf</span><span class="w"> </span><span class="n">tmp</span><span class="p">]</span><span class="err">#</span><span class="w"> </span><span class="n">exit</span>
<span class="n">exit</span>
</code></pre></div>
<p>Since the command is rather long to memorize I created an alias for it in my <a href="https://wiki.archlinux.org/index.php/Zsh">zsh</a> config (~/.zshrc)</p>
<div class="highlight"><pre><span></span><code>alias ansible-playbook-test='docker run -ti --rm -v "$(pwd)":/tmp --workdir="/tmp" williamyeh/ansible:centos7-onbuild ansible-playbook-wrapper'
</code></pre></div>Jenkins docker-pipeline2017-10-30T19:00:00+01:002017-10-30T00:00:00+01:00Jantag:visibilityspots.org,2017-10-30:/jenkins-docker-pipeline.html<p>in a previous blog post I talked about setting up a <a href="https://visibilityspots.org/dockerized-jenkins.html">dockerized jenkins master/slave setup</a> and setting up a <a href="https://visibilityspots.org/nexus-oss-repository-manager.html">private docker registry using nexus</a>.</p>
<p>The next thing on the roadmap was to use this jenkins setup to actually build new docker images for specific software. Before going to the different teams and talking how they now build their software and how this could be done using this new containerized setup I setted up a new jenkins job.</p>
<p>This jenkins job will build a generic jenkins slave docker container which will be used by the jenkins master to build some …</p><p>in a previous blog post I talked about setting up a <a href="https://visibilityspots.org/dockerized-jenkins.html">dockerized jenkins master/slave setup</a> and setting up a <a href="https://visibilityspots.org/nexus-oss-repository-manager.html">private docker registry using nexus</a>.</p>
<p>The next thing on the roadmap was to use this jenkins setup to actually build new docker images for specific software. Before going to the different teams and talking how they now build their software and how this could be done using this new containerized setup I setted up a new jenkins job.</p>
<p>This jenkins job will build a generic jenkins slave docker container which will be used by the jenkins master to build some generic jobs.</p>
<p>to be able to build docker images through jenkins there is the <a href="https://wiki.jenkins.io/display/JENKINS/Docker+Pipeline+Plugin">docker-pipeline</a> plugin which can be used by seeding a repository with the Dockerfile and a Jenkinsfile as described by this <a href="https://getintodevops.com/blog/building-your-first-docker-image-with-jenkins-2-guide-for-developers">tutorial</a>.</p>
<p>To get it configured I had to install the pipeline plugin, configure an SSH key into jenkins and github so jenkins was able to pull the repository together with the docker registry credentials from the private nexus which will be used in the jenkinsfile.</p>
<p>To use the docker-plugin docker needs to be installed on the jenkinsmaster. We already covered that part in the <a href="https://visibilityspots.org/dockerized-jenkins.html">dockerized jenkins master/slave</a> post.</p>
<p>Also the jenkins user needs to be added to the docker group too so it could try to communicate with the docker socket. Which is a weird combination because it's using the docker daemon of the docker node since the socket has been mounted on the jenkins master container.</p>
<p>Because we mount the docker socket from the host to the docker daemon the GID's of the docker group on both host and container need to match to each other. This is explained on the <a href="https://github.com/visibilityspots/dockerfile-jenkins-docker#configuration">github</a> page which resides the config files for the <a href="https://hub.docker.com/r/visibilityspots/jenkins-docker/">jenkins-docker</a> image.</p>
<h2>configuration</h2>
<h3>GID</h3>
<p>The container docker GID is already configured statically to 900 so by changing the one on the host they match and no permission issues should arise concerning this topic.</p>
<h2>credentials</h2>
<h3>jenkins</h3>
<p>To enable jenkins to read from github a jenkins public SSH key need to be added to repository and the private SSH key needs to be configured in jenkins</p>
<p>Next to the repository credentials we also need to configure credentials of the docker repository hosted on our nexus instance and refer to the ID in the jenkinsfile so the pipeline plugin can act accordingly.</p>
<h2>Jenkinsfile</h2>
<p>The steps to be executed are defined in a Jenkinsfile this file can be added in a repository next to the Dockerfile and will perform some stages;</p>
<h3>clone repository</h3>
<ul>
<li>perform a git checkout to the latest update</li>
<li>trim the first 6 chars of the last commit to use as a tag version for the docker image to be builded</li>
</ul>
<h3>build image</h3>
<ul>
<li>build the image based on the Dockerfile</li>
</ul>
<h3>test image</h3>
<ul>
<li>use dgoss to perform a test based on the goss.yaml file</li>
</ul>
<h3>push image</h3>
<ul>
<li>push the created and tested image to the registry with the latest tag and with the short commit hash</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">node</span><span class="w"> </span><span class="p">(</span><span class="s1">'generic'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">def</span><span class="w"> </span><span class="n">container</span>
<span class="w"> </span><span class="n">ansiColor</span><span class="p">(</span><span class="s1">'xterm'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">stage</span><span class="p">(</span><span class="s1">'Clone repository'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">checkout</span><span class="w"> </span><span class="n">scm</span>
<span class="w"> </span><span class="n">shortCommit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sh</span><span class="p">(</span><span class="n">returnStdout</span><span class="p">:</span><span class="w"> </span><span class="bp">true</span><span class="p">,</span><span class="w"> </span><span class="n">script</span><span class="p">:</span><span class="w"> </span><span class="s1">'git rev-parse --short HEAD'</span><span class="p">)</span><span class="o">.</span><span class="n">trim</span><span class="p">()</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">stage</span><span class="p">(</span><span class="s1">'Build image'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">container</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">docker</span><span class="o">.</span><span class="n">build</span><span class="p">(</span><span class="s1">'visibilityspots/jenkins-docker'</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">stage</span><span class="p">(</span><span class="s1">'Test image'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">sh</span><span class="w"> </span><span class="s1">'export GOSS_FILES_STRATEGY=cp && /usr/local/bin/dgoss run --name jenkins-docker-dgoss-test --rm -ti visibilityspots/jenkins-docker'</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">stage</span><span class="p">(</span><span class="s1">'Push image'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">docker</span><span class="o">.</span><span class="n">withRegistry</span><span class="p">(</span><span class="s1">'https://nexus.repository'</span><span class="p">,</span><span class="w"> </span><span class="s1">'nexus-credentials-id'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">container</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="s2">"${shortCommit}"</span><span class="p">)</span>
<span class="w"> </span><span class="n">container</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="s1">'latest'</span><span class="p">)</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h2>testing the docker image</h2>
<p>there is this tool called <a href="https://github.com/aelsabbahy/goss">goss</a> which uses a simple yaml based approach to perform some tests against a server the same way serverspe wcorks. But there is a great wrapper script created called <a href="https://github.com/aelsabbahy/goss/tree/master/extras/dgoss">dgoss</a> which uses the same yaml file to perform the tests against a docker container it spins up especially for the testing case.</p>
<p>When one of the defined tests fails after the image has been build using the Dockerfile, the jenkins pipeline will abort and will not push the image to the registry. To be able to run the goss test suite the software is also preinstalled on our jenkins-slave docker images.</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span><span class="n">jenkins-docker</span><span class="o">]</span><span class="w"> </span><span class="n">Running</span><span class="w"> </span><span class="n">shell</span><span class="w"> </span><span class="n">script</span>
<span class="n">export</span><span class="w"> </span><span class="n">GOSS_FILES_STRATEGY</span><span class="o">=</span><span class="n">cp</span>
<span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="k">local</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">dgoss</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="o">--</span><span class="n">name</span><span class="w"> </span><span class="n">jenkins</span><span class="o">-</span><span class="n">docker</span><span class="o">-</span><span class="n">dgoss</span><span class="o">-</span><span class="n">test</span><span class="w"> </span><span class="o">--</span><span class="n">rm</span><span class="w"> </span><span class="o">-</span><span class="n">ti</span><span class="w"> </span><span class="n">visibilityspots</span><span class="o">/</span><span class="n">jenkins</span><span class="o">-</span><span class="n">docker</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="n">container</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Container</span><span class="w"> </span><span class="nl">ID</span><span class="p">:</span><span class="w"> </span><span class="n">e0849245</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Sleeping</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="mf">0.2</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Running</span><span class="w"> </span><span class="n">Tests</span>
<span class="k">User</span><span class="err">:</span><span class="w"> </span><span class="nl">jenkins</span><span class="p">:</span><span class="w"> </span><span class="ow">exists</span><span class="err">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="k">User</span><span class="err">:</span><span class="w"> </span><span class="nl">jenkins</span><span class="p">:</span><span class="w"> </span><span class="nl">groups</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">["jenkins","docker"</span><span class="o">]</span><span class="err">]</span>
<span class="k">Group</span><span class="err">:</span><span class="w"> </span><span class="nl">docker</span><span class="p">:</span><span class="w"> </span><span class="ow">exists</span><span class="err">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="k">Group</span><span class="err">:</span><span class="w"> </span><span class="nl">docker</span><span class="p">:</span><span class="w"> </span><span class="nl">gid</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">900</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">docker</span><span class="o">-</span><span class="nl">ce</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="k">file</span><span class="err">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Command</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="k">file</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="k">localtime</span><span class="err">:</span><span class="w"> </span><span class="k">exit</span><span class="o">-</span><span class="nl">status</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">0</span><span class="o">]</span>
<span class="nl">Command</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="k">file</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="k">localtime</span><span class="err">:</span><span class="w"> </span><span class="nl">stdout</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">/etc/localtime: symbolic link to /usr/share/zoneinfo/Etc/UTC</span><span class="o">]</span>
<span class="n">Total</span><span class="w"> </span><span class="nl">Duration</span><span class="p">:</span><span class="w"> </span><span class="mf">0.076</span><span class="n">s</span>
<span class="nf">Count</span><span class="err">:</span><span class="w"> </span><span class="mi">8</span><span class="p">,</span><span class="w"> </span><span class="nl">Failed</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nl">Skipped</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Deleting</span><span class="w"> </span><span class="n">container</span>
</code></pre></div>
<h2>jenkins job</h2>
<p>Next up is the configuration of a pipeline defined jenkins job where the only config is the git repository and a pointer to the Jenkinsfile. If everything went well jenkins will execute the steps defined in the Jenkinsfile and store the docker image as a result in the nexus docker repository.</p>
<h3>configuration</h3>
<table>
<thead>
<tr>
<th>Name</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>Poll SCM</td>
<td>*/1 * * * *</td>
</tr>
</tbody>
</table>
<h2>references</h2>
<p><a href="https://getintodevops.com/blog/building-your-first-docker-image-with-jenkins-2-guide-for-developers">https://getintodevops.com/blog/building-your-first-docker-image-with-jenkins-2-guide-for-developers</a></p>Jenkins gradle build succeeded with failure2017-09-21T22:00:00+02:002017-12-21T00:00:00+01:00Jantag:visibilityspots.org,2017-09-21:/gradle-build.html<p>Today we bumped into an interesting issue in the jenkins builds of some android based applications. The gradle commands succeeded but then suddenly failed the build with this most cryptic message ever:</p>
<div class="highlight"><pre><span></span><code><span class="n">BUILD</span><span class="w"> </span><span class="n">SUCCESSFUL</span>
<span class="n">Total</span><span class="w"> </span><span class="nc">time</span><span class="err">:</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">mins</span><span class="w"> </span><span class="mf">20.492</span><span class="w"> </span><span class="n">secs</span>
<span class="nl">FAILURE</span><span class="p">:</span><span class="w"> </span><span class="n">Build</span><span class="w"> </span><span class="n">failed</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="k">exception</span><span class="p">.</span>
<span class="o">*</span><span class="w"> </span><span class="n">What</span><span class="w"> </span><span class="n">went</span><span class="w"> </span><span class="nl">wrong</span><span class="p">:</span>
<span class="n">Already</span><span class="w"> </span><span class="n">finished</span>
<span class="o">*</span><span class="w"> </span><span class="k">Try</span><span class="err">:</span>
<span class="n">Run</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="o">--</span><span class="n">stacktrace</span><span class="w"> </span><span class="k">option</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">get</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">stack</span><span class="w"> </span><span class="n">trace</span><span class="p">.</span><span class="w"> </span><span class="n">Run</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="o">--</span><span class="n">info</span><span class="w"> </span><span class="ow">or</span><span class="w"> </span><span class="o">--</span><span class="n">debug</span><span class="w"> </span><span class="k">option</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">get</span><span class="w"> </span><span class="n">more</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="k">output</span><span class="p">.</span>
<span class="o">[</span><span class="n">Pipeline</span><span class="o">]</span><span class="w"> </span><span class="err">}</span>
</code></pre></div>
<p>Since this came out of nowhere without any modification on the build servers we where flabbergasted since the builds ran fine on our local machines.</p>
<p>So …</p><p>Today we bumped into an interesting issue in the jenkins builds of some android based applications. The gradle commands succeeded but then suddenly failed the build with this most cryptic message ever:</p>
<div class="highlight"><pre><span></span><code><span class="n">BUILD</span><span class="w"> </span><span class="n">SUCCESSFUL</span>
<span class="n">Total</span><span class="w"> </span><span class="nc">time</span><span class="err">:</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">mins</span><span class="w"> </span><span class="mf">20.492</span><span class="w"> </span><span class="n">secs</span>
<span class="nl">FAILURE</span><span class="p">:</span><span class="w"> </span><span class="n">Build</span><span class="w"> </span><span class="n">failed</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="k">exception</span><span class="p">.</span>
<span class="o">*</span><span class="w"> </span><span class="n">What</span><span class="w"> </span><span class="n">went</span><span class="w"> </span><span class="nl">wrong</span><span class="p">:</span>
<span class="n">Already</span><span class="w"> </span><span class="n">finished</span>
<span class="o">*</span><span class="w"> </span><span class="k">Try</span><span class="err">:</span>
<span class="n">Run</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="o">--</span><span class="n">stacktrace</span><span class="w"> </span><span class="k">option</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">get</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">stack</span><span class="w"> </span><span class="n">trace</span><span class="p">.</span><span class="w"> </span><span class="n">Run</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="o">--</span><span class="n">info</span><span class="w"> </span><span class="ow">or</span><span class="w"> </span><span class="o">--</span><span class="n">debug</span><span class="w"> </span><span class="k">option</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">get</span><span class="w"> </span><span class="n">more</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="k">output</span><span class="p">.</span>
<span class="o">[</span><span class="n">Pipeline</span><span class="o">]</span><span class="w"> </span><span class="err">}</span>
</code></pre></div>
<p>Since this came out of nowhere without any modification on the build servers we where flabbergasted since the builds ran fine on our local machines.</p>
<p>So we took the suggestion of the stacktrace option and ran the build again:</p>
<div class="highlight"><pre><span></span><code><span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.371</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="nl">FAILURE</span><span class="p">:</span><span class="w"> </span><span class="n">Build</span><span class="w"> </span><span class="n">failed</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="k">exception</span><span class="p">.</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.372</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.372</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">What</span><span class="w"> </span><span class="n">went</span><span class="w"> </span><span class="nl">wrong</span><span class="p">:</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.372</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="n">Already</span><span class="w"> </span><span class="n">finished</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.372</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.372</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">Exception</span><span class="w"> </span><span class="k">is</span><span class="err">:</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.373</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="n">lang</span><span class="p">.</span><span class="nl">IllegalStateException</span><span class="p">:</span><span class="w"> </span><span class="n">Already</span><span class="w"> </span><span class="n">finished</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.373</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="n">google</span><span class="p">.</span><span class="n">common</span><span class="p">.</span><span class="n">base</span><span class="p">.</span><span class="n">Preconditions</span><span class="p">.</span><span class="n">checkState</span><span class="p">(</span><span class="n">Preconditions</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">174</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="n">android</span><span class="p">.</span><span class="n">builder</span><span class="p">.</span><span class="n">profile</span><span class="p">.</span><span class="n">ProcessProfileWriter</span><span class="p">.</span><span class="n">finishAndMaybeWrite</span><span class="p">(</span><span class="n">ProcessProfileWriter</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">121</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="n">android</span><span class="p">.</span><span class="n">builder</span><span class="p">.</span><span class="n">profile</span><span class="p">.</span><span class="n">ProcessProfileWriterFactory</span><span class="p">.</span><span class="n">shutdownAndMaybeWrite</span><span class="p">(</span><span class="n">ProcessProfileWriterFactory</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">52</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="n">android</span><span class="p">.</span><span class="n">build</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">profile</span><span class="p">.</span><span class="n">ProfilerInitializer</span><span class="err">$</span><span class="n">ProfileShutdownListener</span><span class="p">.</span><span class="n">completed</span><span class="p">(</span><span class="n">ProfilerInitializer</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">110</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="n">reflect</span><span class="p">.</span><span class="n">GeneratedMethodAccessor660</span><span class="p">.</span><span class="n">invoke</span><span class="p">(</span><span class="k">Unknown</span><span class="w"> </span><span class="n">Source</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">sun</span><span class="p">.</span><span class="n">reflect</span><span class="p">.</span><span class="n">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="n">invoke</span><span class="p">(</span><span class="n">DelegatingMethodAccessorImpl</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">43</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="n">lang</span><span class="p">.</span><span class="n">reflect</span><span class="p">.</span><span class="k">Method</span><span class="p">.</span><span class="n">invoke</span><span class="p">(</span><span class="k">Method</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">498</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">dispatch</span><span class="p">.</span><span class="n">ReflectionDispatch</span><span class="p">.</span><span class="n">dispatch</span><span class="p">(</span><span class="n">ReflectionDispatch</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">35</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">dispatch</span><span class="p">.</span><span class="n">ReflectionDispatch</span><span class="p">.</span><span class="n">dispatch</span><span class="p">(</span><span class="n">ReflectionDispatch</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">24</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">event</span><span class="p">.</span><span class="n">DefaultListenerManager</span><span class="err">$</span><span class="n">ListenerDetails</span><span class="p">.</span><span class="n">dispatch</span><span class="p">(</span><span class="n">DefaultListenerManager</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">249</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">event</span><span class="p">.</span><span class="n">DefaultListenerManager</span><span class="err">$</span><span class="n">ListenerDetails</span><span class="p">.</span><span class="n">dispatch</span><span class="p">(</span><span class="n">DefaultListenerManager</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">229</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">event</span><span class="p">.</span><span class="n">AbstractBroadcastDispatch</span><span class="p">.</span><span class="n">dispatch</span><span class="p">(</span><span class="n">AbstractBroadcastDispatch</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">44</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">event</span><span class="p">.</span><span class="n">DefaultListenerManager</span><span class="err">$</span><span class="n">EventBroadcast</span><span class="err">$</span><span class="n">ListenerDispatch</span><span class="p">.</span><span class="n">dispatch</span><span class="p">(</span><span class="n">DefaultListenerManager</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">221</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.374</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">event</span><span class="p">.</span><span class="n">DefaultListenerManager</span><span class="err">$</span><span class="n">EventBroadcast</span><span class="err">$</span><span class="n">ListenerDispatch</span><span class="p">.</span><span class="n">dispatch</span><span class="p">(</span><span class="n">DefaultListenerManager</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">209</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">dispatch</span><span class="p">.</span><span class="n">ProxyDispatchAdapter</span><span class="err">$</span><span class="n">DispatchingInvocationHandler</span><span class="p">.</span><span class="n">invoke</span><span class="p">(</span><span class="n">ProxyDispatchAdapter</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">93</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">com</span><span class="p">.</span><span class="n">sun</span><span class="p">.</span><span class="n">proxy</span><span class="p">.</span><span class="err">$</span><span class="n">Proxy17</span><span class="p">.</span><span class="n">completed</span><span class="p">(</span><span class="k">Unknown</span><span class="w"> </span><span class="n">Source</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">initialization</span><span class="p">.</span><span class="n">DefaultGradleLauncher</span><span class="p">.</span><span class="n">stop</span><span class="p">(</span><span class="n">DefaultGradleLauncher</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">226</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">InProcessBuildActionExecuter</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">InProcessBuildActionExecuter</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">44</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">InProcessBuildActionExecuter</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">InProcessBuildActionExecuter</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">26</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">tooling</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">provider</span><span class="p">.</span><span class="n">ContinuousBuildActionExecuter</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">ContinuousBuildActionExecuter</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">75</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">tooling</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">provider</span><span class="p">.</span><span class="n">ContinuousBuildActionExecuter</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">ContinuousBuildActionExecuter</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">49</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">tooling</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">provider</span><span class="p">.</span><span class="n">ServicesSetupBuildActionExecuter</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">ServicesSetupBuildActionExecuter</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">49</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">tooling</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">provider</span><span class="p">.</span><span class="n">ServicesSetupBuildActionExecuter</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">ServicesSetupBuildActionExecuter</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">31</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.375</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">ExecuteBuild</span><span class="p">.</span><span class="n">doBuild</span><span class="p">(</span><span class="n">ExecuteBuild</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">67</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">BuildCommandOnly</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">BuildCommandOnly</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">36</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="n">proceed</span><span class="p">(</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">120</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">WatchForDisconnection</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">WatchForDisconnection</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">47</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="n">proceed</span><span class="p">(</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">120</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">ResetDeprecationLogger</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">ResetDeprecationLogger</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">26</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="n">proceed</span><span class="p">(</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">120</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">RequestStopIfSingleUsedDaemon</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">RequestStopIfSingleUsedDaemon</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">34</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="n">proceed</span><span class="p">(</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">120</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">ForwardClientInput</span><span class="err">$</span><span class="mf">2.</span><span class="k">call</span><span class="p">(</span><span class="n">ForwardClientInput</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">74</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">ForwardClientInput</span><span class="err">$</span><span class="mf">2.</span><span class="k">call</span><span class="p">(</span><span class="n">ForwardClientInput</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">72</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">util</span><span class="p">.</span><span class="n">Swapper</span><span class="p">.</span><span class="n">swap</span><span class="p">(</span><span class="n">Swapper</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">38</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">ForwardClientInput</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">ForwardClientInput</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">72</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="n">proceed</span><span class="p">(</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">120</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.376</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">LogAndCheckHealth</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">LogAndCheckHealth</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">55</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="n">proceed</span><span class="p">(</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">120</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">LogToClient</span><span class="p">.</span><span class="n">doBuild</span><span class="p">(</span><span class="n">LogToClient</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">60</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">BuildCommandOnly</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">BuildCommandOnly</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">36</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="n">proceed</span><span class="p">(</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">120</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">EstablishBuildEnvironment</span><span class="p">.</span><span class="n">doBuild</span><span class="p">(</span><span class="n">EstablishBuildEnvironment</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">72</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">BuildCommandOnly</span><span class="p">.</span><span class="k">execute</span><span class="p">(</span><span class="n">BuildCommandOnly</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">36</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">api</span><span class="p">.</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="n">proceed</span><span class="p">(</span><span class="n">DaemonCommandExecution</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">120</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="k">exec</span><span class="p">.</span><span class="n">StartBuildOrRespondWithBusy</span><span class="err">$</span><span class="mf">1.</span><span class="n">run</span><span class="p">(</span><span class="n">StartBuildOrRespondWithBusy</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">50</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">launcher</span><span class="p">.</span><span class="n">daemon</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">DaemonStateCoordinator</span><span class="err">$</span><span class="mf">1.</span><span class="n">run</span><span class="p">(</span><span class="n">DaemonStateCoordinator</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">297</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">concurrent</span><span class="p">.</span><span class="n">ExecutorPolicy</span><span class="err">$</span><span class="n">CatchAndRecordFailures</span><span class="p">.</span><span class="n">onExecute</span><span class="p">(</span><span class="n">ExecutorPolicy</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">63</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">org</span><span class="p">.</span><span class="n">gradle</span><span class="p">.</span><span class="n">internal</span><span class="p">.</span><span class="n">concurrent</span><span class="p">.</span><span class="n">StoppableExecutorImpl</span><span class="err">$</span><span class="mf">1.</span><span class="n">run</span><span class="p">(</span><span class="n">StoppableExecutorImpl</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">46</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="n">util</span><span class="p">.</span><span class="n">concurrent</span><span class="p">.</span><span class="n">ThreadPoolExecutor</span><span class="p">.</span><span class="n">runWorker</span><span class="p">(</span><span class="n">ThreadPoolExecutor</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">1142</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.377</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="n">util</span><span class="p">.</span><span class="n">concurrent</span><span class="p">.</span><span class="n">ThreadPoolExecutor</span><span class="err">$</span><span class="n">Worker</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">ThreadPoolExecutor</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">617</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.378</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span><span class="w"> </span><span class="k">at</span><span class="w"> </span><span class="n">java</span><span class="p">.</span><span class="n">lang</span><span class="p">.</span><span class="n">Thread</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">Thread</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">745</span><span class="p">)</span>
<span class="mi">14</span><span class="err">:</span><span class="mi">00</span><span class="err">:</span><span class="mf">31.378</span><span class="w"> </span><span class="o">[</span><span class="n">ERROR</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">org.gradle.internal.buildevents.BuildExceptionReporter</span><span class="o">]</span>
</code></pre></div>
<p>but this didn't make us giving any more clue. So decided to log on to the build node and trying to remove the gradle caches on suggestion of a colleague:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rm<span class="w"> </span>/home/jenkins/.gradle/caches/*<span class="w"> </span>-rf
</code></pre></div>
<p>but no solution neither. Checked out the repository to commit of the latest successful build and tried to run the command from there manually but still no success.</p>
<p>After reading many other gradle issues I came to one indicating killing the gradle daemons could solve the mystery. So we went ahead and killed the daemons on the build node.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># ps aux | grep gradle</span>
<span class="n">jenkins</span><span class="w"> </span><span class="mi">6341</span><span class="w"> </span><span class="mf">33.9</span><span class="w"> </span><span class="mf">7.9</span><span class="w"> </span><span class="mi">14954276</span><span class="w"> </span><span class="mi">2625892</span><span class="w"> </span><span class="err">?</span><span class="w"> </span><span class="n">Ssl</span><span class="w"> </span><span class="mi">18</span><span class="p">:</span><span class="mi">11</span><span class="w"> </span><span class="mi">58</span><span class="p">:</span><span class="mi">59</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">jvm</span><span class="o">/</span><span class="n">java</span><span class="o">-</span><span class="mi">8</span><span class="o">-</span><span class="n">openjdk</span><span class="o">-</span><span class="n">amd64</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">java</span><span class="w"> </span><span class="o">-</span><span class="n">Xmx1536m</span><span class="w"> </span><span class="o">-</span><span class="n">Dfile</span><span class="o">.</span><span class="n">encoding</span><span class="o">=</span><span class="n">UTF</span><span class="o">-</span><span class="mi">8</span><span class="w"> </span><span class="o">-</span><span class="n">Duser</span><span class="o">.</span><span class="n">country</span><span class="o">=</span><span class="n">US</span><span class="w"> </span><span class="o">-</span><span class="n">Duser</span><span class="o">.</span><span class="n">language</span><span class="o">=</span><span class="n">en</span><span class="w"> </span><span class="o">-</span><span class="n">Duser</span><span class="o">.</span><span class="n">variant</span><span class="w"> </span><span class="o">-</span><span class="n">cp</span><span class="w"> </span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">jenkins</span><span class="o">/.</span><span class="n">gradle</span><span class="o">/</span><span class="n">wrapper</span><span class="o">/</span><span class="n">dists</span><span class="o">/</span><span class="n">gradle</span><span class="o">-</span><span class="mf">3.5</span><span class="o">-</span><span class="n">all</span><span class="o">/</span><span class="n">exhrs6ca08n232b14ue48lbye</span><span class="o">/</span><span class="n">gradle</span><span class="o">-</span><span class="mf">3.5</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">gradle</span><span class="o">-</span><span class="n">launcher</span><span class="o">-</span><span class="mf">3.5</span><span class="o">.</span><span class="n">jar</span><span class="w"> </span><span class="n">org</span><span class="o">.</span><span class="n">gradle</span><span class="o">.</span><span class="n">launcher</span><span class="o">.</span><span class="n">daemon</span><span class="o">.</span><span class="n">bootstrap</span><span class="o">.</span><span class="n">GradleDaemon</span><span class="w"> </span><span class="mf">3.5</span>
<span class="n">jenkins</span><span class="w"> </span><span class="mi">10631</span><span class="w"> </span><span class="mf">19.3</span><span class="w"> </span><span class="mf">6.5</span><span class="w"> </span><span class="mi">14449300</span><span class="w"> </span><span class="mi">2164472</span><span class="w"> </span><span class="err">?</span><span class="w"> </span><span class="n">Ssl</span><span class="w"> </span><span class="mi">18</span><span class="p">:</span><span class="mi">48</span><span class="w"> </span><span class="mi">26</span><span class="p">:</span><span class="mi">25</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">jvm</span><span class="o">/</span><span class="n">java</span><span class="o">-</span><span class="mi">8</span><span class="o">-</span><span class="n">openjdk</span><span class="o">-</span><span class="n">amd64</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">java</span><span class="w"> </span><span class="o">-</span><span class="n">Xmx1536m</span><span class="w"> </span><span class="o">-</span><span class="n">Dfile</span><span class="o">.</span><span class="n">encoding</span><span class="o">=</span><span class="n">UTF</span><span class="o">-</span><span class="mi">8</span><span class="w"> </span><span class="o">-</span><span class="n">Duser</span><span class="o">.</span><span class="n">country</span><span class="o">=</span><span class="n">US</span><span class="w"> </span><span class="o">-</span><span class="n">Duser</span><span class="o">.</span><span class="n">language</span><span class="o">=</span><span class="n">en</span><span class="w"> </span><span class="o">-</span><span class="n">Duser</span><span class="o">.</span><span class="n">variant</span><span class="w"> </span><span class="o">-</span><span class="n">cp</span><span class="w"> </span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">jenkins</span><span class="o">/.</span><span class="n">gradle</span><span class="o">/</span><span class="n">wrapper</span><span class="o">/</span><span class="n">dists</span><span class="o">/</span><span class="n">gradle</span><span class="o">-</span><span class="mf">3.4</span><span class="o">.</span><span class="mi">1</span><span class="o">-</span><span class="n">all</span><span class="o">/</span><span class="n">c3ib5obfnqr0no9szq6qc17do</span><span class="o">/</span><span class="n">gradle</span><span class="o">-</span><span class="mf">3.4</span><span class="o">.</span><span class="mi">1</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">gradle</span><span class="o">-</span><span class="n">launcher</span><span class="o">-</span><span class="mf">3.4</span><span class="o">.</span><span class="mf">1.j</span><span class="n">ar</span><span class="w"> </span><span class="n">org</span><span class="o">.</span><span class="n">gradle</span><span class="o">.</span><span class="n">launcher</span><span class="o">.</span><span class="n">daemon</span><span class="o">.</span><span class="n">bootstrap</span><span class="o">.</span><span class="n">GradleDaemon</span><span class="w"> </span><span class="mf">3.4</span><span class="o">.</span><span class="mi">1</span>
<span class="n">kill</span><span class="w"> </span><span class="o">-</span><span class="mi">9</span><span class="w"> </span><span class="mi">6341</span><span class="w"> </span><span class="mi">10631</span>
</code></pre></div>
<p>And reran the build manually which succeeded, when re-triggering the job it succeeded. Victory! We assumed those daemons where stopped right after the execution but as we noticed this wasn't the case.</p>
<p>I wrote this post since I couldn't find anything related to the message and maybe I can help others gaining time resolving the issue since as always such things comes on days you haven't time for them..</p>Nexus OSS repository manager2017-09-11T19:00:00+02:002017-09-11T00:00:00+02:00Jantag:visibilityspots.org,2017-09-11:/nexus-oss-repository-manager.html<p>looking for a global repository store which could store maven projects, yum repositories, docker repositories, we bumped into <a href="https://help.sonatype.com/display/NXRM3/Repository+Manager+3">Nexus repository manager</a>. We used the official docker image to see how it can be implemented in the dockerized CI environment.</p>
<h2>docker repository</h2>
<p>as a first the docker repository feature could be enabled so we can start building and storing docker images for the different jenkins build slaves and the jenkins master so our work is reproducible and stored in a safe central place.</p>
<p>We configured 3 repositories in nexus for our docker images seen as a recommended approach in the <a href="https://help.sonatype.com/display/NXRM3/Private+Registry+for+Docker#PrivateRegistryforDocker-HostedRepositoryforDocker(PrivateRegistryforDocker)">nexus …</a></p><p>looking for a global repository store which could store maven projects, yum repositories, docker repositories, we bumped into <a href="https://help.sonatype.com/display/NXRM3/Repository+Manager+3">Nexus repository manager</a>. We used the official docker image to see how it can be implemented in the dockerized CI environment.</p>
<h2>docker repository</h2>
<p>as a first the docker repository feature could be enabled so we can start building and storing docker images for the different jenkins build slaves and the jenkins master so our work is reproducible and stored in a safe central place.</p>
<p>We configured 3 repositories in nexus for our docker images seen as a recommended approach in the <a href="https://help.sonatype.com/display/NXRM3/Private+Registry+for+Docker#PrivateRegistryforDocker-HostedRepositoryforDocker(PrivateRegistryforDocker)">nexus documentation</a>. Each of them are configured to their own separate blob store.</p>
<table>
<thead>
<tr>
<th>Name</th>
<th></th>
<th>Purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td>private</td>
<td></td>
<td>self hosted docker repository where all the internal images will be stored</td>
</tr>
<tr>
<td>proxy</td>
<td></td>
<td>cached proxy which will download every request from docker hub and caches to reduce download but also for having an offline backup of upstream images</td>
</tr>
<tr>
<td>group</td>
<td></td>
<td>group which combines the first 2 repositories behind one URL</td>
</tr>
</tbody>
</table>
<h3>configuration</h3>
<p>by following a tutorial from <a href="https://www.ivankrizsan.se/2016/06/09/create-a-private-docker-registry/">Ivan Krizan</a> a working example has been configured, but this in a way without https enabled which is not the recommended manner to set it up. Therefore we looked into enabling https instead. This can be done in different ways. </p>
<h4>SSL directly configured</h4>
<p>setting it up with https doesn't seemed straightforward with nexus directly connected. Also it has a disadvantage in the docker world that the certificates needs to be present on the container and some configuration changes needs to be made. This makes upgrading the nexus container on it's own a bit of a more complex task as without the direct SSL encryption enabled.</p>
<h4>reverse proxy setup</h4>
<p>we could instead spin up an nginx reverse proxy which will handle the encrypted requests towards the client. In the back it communicates to the nexus container service through plain HTTP on a dedicated docker network only for this kind of traffic.</p>
<p>Based on the standard setup for one private repository by <a href="https://stefanprodan.com/2016/docker-private-registry-nexus-nginx/">Stefan Prodan</a> I came to the described nginx configuration.</p>
<p>It will serve the web GUI when you access the proxy using a normal browser by a redirect to the nexus container on port 8081 where the management console is living. </p>
<p>For the docker client it will act differently, when a docker pull command is executed it will get redirected to the docker-group repository which combines both images from upstream (cached) as well as images from the private docker repository.</p>
<p>On the other hand when issuing a docker push command the image will get pushed towards the private repository only since group and proxy repositories aren't able to do so.</p>
<div class="highlight"><pre><span></span><code><span class="n">worker_processes</span><span class="w"> </span><span class="mi">2</span><span class="p">;</span>
<span class="n">events</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">worker_connections</span><span class="w"> </span><span class="mi">1024</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">http</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">error_log</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="nb">log</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">error</span><span class="o">.</span><span class="n">log</span><span class="w"> </span><span class="n">warn</span><span class="p">;</span>
<span class="w"> </span><span class="n">access_log</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="nb nb-Type">null</span><span class="p">;</span>
<span class="w"> </span><span class="n">proxy_intercept_errors</span><span class="w"> </span><span class="n">off</span><span class="p">;</span>
<span class="w"> </span><span class="n">proxy_send_timeout</span><span class="w"> </span><span class="mi">120</span><span class="p">;</span>
<span class="w"> </span><span class="n">proxy_read_timeout</span><span class="w"> </span><span class="mi">300</span><span class="p">;</span>
<span class="w"> </span><span class="n">upstream</span><span class="w"> </span><span class="n">nexus</span><span class="o">-</span><span class="n">gui</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">server</span><span class="w"> </span><span class="n">nexus</span><span class="p">:</span><span class="mi">8081</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">upstream</span><span class="w"> </span><span class="n">docker</span><span class="o">-</span><span class="n">private</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">server</span><span class="w"> </span><span class="n">nexus</span><span class="p">:</span><span class="mi">5000</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">upstream</span><span class="w"> </span><span class="n">docker</span><span class="o">-</span><span class="n">group</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">server</span><span class="w"> </span><span class="n">nexus</span><span class="p">:</span><span class="mi">5001</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="o">$</span><span class="n">request_method</span><span class="w"> </span><span class="o">$</span><span class="n">redirection</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">default</span><span class="w"> </span><span class="s2">"nexus-gui"</span><span class="p">;</span>
<span class="w"> </span><span class="s2">"~GET"</span><span class="w"> </span><span class="s2">"docker-group"</span><span class="p">;</span>
<span class="w"> </span><span class="s2">"~(HEAD|PATCH|PUT|POST|DELETE)"</span><span class="w"> </span><span class="s2">"docker-private"</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">server</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">listen</span><span class="w"> </span><span class="mi">80</span><span class="p">;</span>
<span class="w"> </span><span class="n">listen</span><span class="w"> </span><span class="mi">443</span><span class="w"> </span><span class="n">ssl</span><span class="p">;</span>
<span class="w"> </span><span class="n">server_name</span><span class="w"> </span><span class="n">nexus</span><span class="o">.</span><span class="n">domain</span><span class="o">.</span><span class="n">org</span><span class="p">;</span>
<span class="w"> </span><span class="n">ssl_certificate</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">nginx</span><span class="o">.</span><span class="n">crt</span><span class="p">;</span>
<span class="w"> </span><span class="n">ssl_certificate_key</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">nginx</span><span class="o">/</span><span class="n">nginx</span><span class="o">.</span><span class="n">key</span><span class="p">;</span>
<span class="w"> </span><span class="n">keepalive_timeout</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="mi">5</span><span class="p">;</span>
<span class="w"> </span><span class="n">proxy_buffering</span><span class="w"> </span><span class="n">off</span><span class="p">;</span>
<span class="w"> </span><span class="c1"># allow large uploads</span>
<span class="w"> </span><span class="n">client_max_body_size</span><span class="w"> </span><span class="mi">1</span><span class="n">G</span><span class="p">;</span>
<span class="w"> </span><span class="n">location</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">$</span><span class="n">http_user_agent</span><span class="w"> </span><span class="o">~</span><span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">proxy_pass</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//$</span><span class="n">redirection</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">proxy_pass</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">nexus</span><span class="o">-</span><span class="n">gui</span><span class="p">;</span>
<span class="w"> </span><span class="n">proxy_set_header</span><span class="w"> </span><span class="n">Host</span><span class="w"> </span><span class="o">$</span><span class="n">host</span><span class="p">;</span>
<span class="w"> </span><span class="n">proxy_set_header</span><span class="w"> </span><span class="n">X</span><span class="o">-</span><span class="n">Real</span><span class="o">-</span><span class="n">IP</span><span class="w"> </span><span class="o">$</span><span class="n">remote_addr</span><span class="p">;</span>
<span class="w"> </span><span class="n">proxy_set_header</span><span class="w"> </span><span class="n">X</span><span class="o">-</span><span class="n">Forwarded</span><span class="o">-</span><span class="n">For</span><span class="w"> </span><span class="o">$</span><span class="n">proxy_add_x_forwarded_for</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Unfortunately when using a self signed key pair the client will not connect properly to the repository. You'll need to use <a href="https://letsencrypt.org">letsencrypt</a> or another party to get a valid certificate from;</p>
<div class="highlight"><pre><span></span><code><span class="o">[</span><span class="n">root@dockernode ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="n">login</span><span class="w"> </span><span class="n">nexus</span><span class="p">.</span><span class="k">domain</span><span class="p">.</span><span class="n">org</span>
<span class="nl">Username</span><span class="p">:</span><span class="w"> </span><span class="n">test</span>
<span class="nl">Password</span><span class="p">:</span>
<span class="n">Error</span><span class="w"> </span><span class="n">response</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="nl">daemon</span><span class="p">:</span><span class="w"> </span><span class="k">Get</span><span class="w"> </span><span class="nl">https</span><span class="p">:</span><span class="o">//</span><span class="n">nexus</span><span class="p">.</span><span class="k">domain</span><span class="p">.</span><span class="n">org</span><span class="o">/</span><span class="n">v2</span><span class="o">/</span><span class="err">:</span><span class="w"> </span><span class="nl">x509</span><span class="p">:</span><span class="w"> </span><span class="n">certificate</span><span class="w"> </span><span class="n">signed</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="k">unknown</span><span class="w"> </span><span class="n">authority</span>
</code></pre></div>
<h4>letsencrypt setup</h4>
<p>when looking for solutions <a href="https://letsencrypt.org">letsencrypt</a> could be an option in combination with an <a href="https://github.com/jwilder/nginx-proxy">nginx-proxy</a> container. But for the moment we placed the auto nginx-proxy on a side track since only one service is needing the proxy and it can perfectly be combined with a docker-compose setup.</p>
<h4>self signed certificates</h4>
<p>in the meanwhile to gain some time we'll stick with the self generated scripts and use the insecure-registry workaround on the docker nodes, jenkins build servers and my local client. When heading for production we'll generate certificates with letsencrypt and I'll update this post.</p>
<h5>centos 7 docker node daemon configuration</h5>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="n">root</span><span class="err">@</span><span class="n">dockernode</span><span class="w"> </span><span class="o">~</span><span class="p">]</span><span class="c1"># vim /etc/systemd/system/docker.service.d/docker.conf</span>
<span class="p">[</span><span class="n">Service</span><span class="p">]</span>
<span class="n">ExecStart</span><span class="o">=</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">dockerd</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="n">tcp</span><span class="p">:</span><span class="o">//</span><span class="mf">0.0</span><span class="o">.</span><span class="mf">0.0</span><span class="p">:</span><span class="mi">2375</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="n">unix</span><span class="p">:</span><span class="o">//</span><span class="k">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">docker</span><span class="o">.</span><span class="n">sock</span><span class="w"> </span><span class="o">--</span><span class="n">insecure</span><span class="o">-</span><span class="n">registry</span><span class="o">=</span><span class="n">nexus</span><span class="o">.</span><span class="n">domain</span><span class="o">.</span><span class="n">org</span>
<span class="p">[</span><span class="n">root</span><span class="err">@</span><span class="n">dockernode</span><span class="w"> </span><span class="o">~</span><span class="p">]</span><span class="c1"># systemctl daemon-reload</span>
<span class="p">[</span><span class="n">root</span><span class="err">@</span><span class="n">dockernode</span><span class="w"> </span><span class="o">~</span><span class="p">]</span><span class="c1"># systemctl restart docker.service</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="o">[</span><span class="n">root@dockernode ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="n">login</span><span class="w"> </span><span class="n">nexus</span><span class="p">.</span><span class="k">domain</span><span class="p">.</span><span class="n">org</span>
<span class="nl">Username</span><span class="p">:</span><span class="w"> </span><span class="n">test</span>
<span class="nl">Password</span><span class="p">:</span>
<span class="n">Login</span><span class="w"> </span><span class="n">Succeeded</span>
</code></pre></div>
<h5>archlinux client configuration</h5>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>cat<span class="w"> </span>/etc/docker/daemon.json
<span class="o">{</span>
<span class="w"> </span><span class="s2">"insecure-registries"</span><span class="w"> </span>:<span class="w"> </span><span class="o">[</span><span class="s2">"nexus.domain.org"</span><span class="o">]</span>
<span class="o">}</span>
$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>restart<span class="w"> </span>docker.service
$<span class="w"> </span>docker<span class="w"> </span>login<span class="w"> </span>nexus.domain.org
Username:<span class="w"> </span><span class="nb">test</span>
Password:
Login<span class="w"> </span>Succeeded
</code></pre></div>
<h2>tips and tricks</h2>
<h3>blobstore</h3>
<p>we created a separate blob store for every repository, that way on the filesystem they are easily recognized for maintenance tasks in the future.</p>
<h3>removal</h3>
<p>when removing a blob store and a repository through the GUI I noticed the actual content on the file system is still available, when recreating the repository and blob store with the same name the content is available again. When removing the content on the filesystem too you have again a clean repository to start with.</p>
<h2>future improvements and features</h2>
<p>there are some features and configuration options that will need to be implemented in the future, when I got there I will update this blog post accordingly.</p>Dockerized jenkins master/slave setup2017-09-06T22:00:00+02:002017-09-06T00:00:00+02:00Jantag:visibilityspots.org,2017-09-06:/dockerized-jenkins.html<p>started at a new customer we were looking for a more flexible way of having jenkins spinning up slaves on the fly. This in a way a slave is only started and consuming resources when a specific job is running. That way those resources could be used more efficient.</p>
<p>Also the fact that developers could take control over their build servers by managing the Dockerfiles themselves is a great advantage too. But that's for a later phase. Let's start at the beginning.</p>
<p>For the docker host a CentOS 7 server has been provisioned and prepared to run the docker daemon …</p><p>started at a new customer we were looking for a more flexible way of having jenkins spinning up slaves on the fly. This in a way a slave is only started and consuming resources when a specific job is running. That way those resources could be used more efficient.</p>
<p>Also the fact that developers could take control over their build servers by managing the Dockerfiles themselves is a great advantage too. But that's for a later phase. Let's start at the beginning.</p>
<p>For the docker host a CentOS 7 server has been provisioned and prepared to run the docker daemon. Starting with updating the OS, removing unnecessary services and implementing NTP.</p>
<div class="highlight"><pre><span></span><code># yum upgrade -y
# systemctl stop postfix
# systemctl disable postfix
# yum remove postfix
# systemctl stop chronyd
# yum remove chrony -y
# reboot
# yum install ntp
# systemctl start ntpd
# systemctl enable ntpd
# ntpdate
# date
</code></pre></div>
<p>Once the system has been prepared we can start with the installation of the docker daemon using the upstream <a href="https://www.docker.com/community-edition">docker community edition</a> repository</p>
<div class="highlight"><pre><span></span><code><span class="c1"># yum install -y yum-utils device-mapper-persistent-data lvm2</span>
<span class="c1"># yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo</span>
<span class="c1"># yum clean all</span>
<span class="c1"># yum makecache fast</span>
<span class="c1"># yum install docker-ce</span>
<span class="c1"># systemctl start docker</span>
<span class="c1"># systemctl enable docker</span>
<span class="c1"># docker run hello-world</span>
<span class="n">Hello</span><span class="w"> </span><span class="n">from</span><span class="w"> </span><span class="n">Docker</span><span class="o">!</span>
<span class="n">This</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="n">shows</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">your</span><span class="w"> </span><span class="n">installation</span><span class="w"> </span><span class="n">appears</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">working</span><span class="w"> </span><span class="n">correctly</span><span class="o">.</span>
<span class="o">...</span>
<span class="c1"># docker info</span>
</code></pre></div>
<p>we do now have a basic docker daemon running on a CentOS 7 machine. The next thing to do is to start a jenkins master docker container. Before doing so some decisions need to be made.</p>
<p>Docker containers are stateless, meaning when the container is gone, the data is gone too. With this taken into account and assuming it's a proof of concept I decided to mount a directory on the host to the jenkins master container.</p>
<p>Since the pipeline plugin will be used, all job configurations are residing in Jenkinsfiles. The needed plugins can be passed through the docker run command later on so that's covered as well.</p>
<p>When the setup is ready for production we could look which directories are needed to be persistent and decide to mount only those needed. But for starters we go for a full mount of the jenkins home directory.</p>
<p>Therefore we need to create a jenkins user on the docker host which is the owner of the local directory which will be used afterwards to mount onto the jenkins master docker container.</p>
<div class="highlight"><pre><span></span><code># mkdir /opt/jenkins
# chown -R 1000: /opt/jenkins
</code></pre></div>
<p>before we can start using this very same docker host through our jenkins instance we need to open up the API port of our docker daemon. This can be configured in the systemd entity of the docker daemon:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># mkdir -p /etc/systemd/system/docker.service.d/</span>
<span class="c1"># cat /etc/systemd/system/docker.service.d/docker.conf</span>
<span class="p">[</span><span class="n">Service</span><span class="p">]</span>
<span class="n">ExecStart</span><span class="o">=</span>
<span class="n">ExecStart</span><span class="o">=/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">dockerd</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="n">tcp</span><span class="p">:</span><span class="o">//</span><span class="mf">0.0</span><span class="o">.</span><span class="mf">0.0</span><span class="p">:</span><span class="mi">2375</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="n">unix</span><span class="p">:</span><span class="o">//</span><span class="k">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">docker</span><span class="o">.</span><span class="n">sock</span>
<span class="c1"># systemctl daemon-reload</span>
<span class="c1"># systemctl restart docker.service</span>
<span class="c1"># systemctl status docker.service</span>
<span class="o">*</span><span class="w"> </span><span class="n">docker</span><span class="o">.</span><span class="n">service</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">Docker</span><span class="w"> </span><span class="n">Application</span><span class="w"> </span><span class="n">Container</span><span class="w"> </span><span class="n">Engine</span>
<span class="n">Loaded</span><span class="p">:</span><span class="w"> </span><span class="n">loaded</span><span class="w"> </span><span class="p">(</span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">systemd</span><span class="o">/</span><span class="n">system</span><span class="o">/</span><span class="n">docker</span><span class="o">.</span><span class="n">service</span><span class="p">;</span><span class="w"> </span><span class="n">enabled</span><span class="p">;</span><span class="w"> </span><span class="n">vendor</span><span class="w"> </span><span class="n">preset</span><span class="p">:</span><span class="w"> </span><span class="n">disabled</span><span class="p">)</span>
<span class="n">Drop</span><span class="o">-</span><span class="n">In</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">systemd</span><span class="o">/</span><span class="n">system</span><span class="o">/</span><span class="n">docker</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">d</span>
<span class="o">-</span><span class="w"> </span><span class="n">docker</span><span class="o">.</span><span class="n">conf</span>
<span class="n">Active</span><span class="p">:</span><span class="w"> </span><span class="n">active</span><span class="w"> </span><span class="p">(</span><span class="n">running</span><span class="p">)</span><span class="w"> </span><span class="n">since</span><span class="w"> </span><span class="n">Thu</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">20</span><span class="w"> </span><span class="mi">13</span><span class="p">:</span><span class="mi">43</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="n">CEST</span><span class="p">;</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">weeks</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="n">days</span><span class="w"> </span><span class="n">ago</span>
<span class="n">Docs</span><span class="p">:</span><span class="w"> </span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">docs</span><span class="o">.</span><span class="n">docker</span><span class="o">.</span><span class="n">com</span>
<span class="n">Main</span><span class="w"> </span><span class="n">PID</span><span class="p">:</span><span class="w"> </span><span class="mi">5858</span><span class="w"> </span><span class="p">(</span><span class="n">dockerd</span><span class="p">)</span>
<span class="n">Memory</span><span class="p">:</span><span class="w"> </span><span class="mf">4.2</span><span class="n">G</span>
<span class="n">CGroup</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">system</span><span class="o">.</span><span class="n">slice</span><span class="o">/</span><span class="n">docker</span><span class="o">.</span><span class="n">service</span>
<span class="o">|-</span><span class="w"> </span><span class="mi">5858</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">dockerd</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="n">tcp</span><span class="p">:</span><span class="o">//</span><span class="mf">0.0</span><span class="o">.</span><span class="mf">0.0</span><span class="p">:</span><span class="mi">2375</span><span class="w"> </span><span class="o">-</span><span class="n">H</span><span class="w"> </span><span class="n">unix</span><span class="p">:</span><span class="o">//</span><span class="k">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">docker</span><span class="o">.</span><span class="n">sock</span>
<span class="o">|-</span><span class="w"> </span><span class="mi">5866</span><span class="w"> </span><span class="n">docker</span><span class="o">-</span><span class="n">containerd</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="n">unix</span><span class="p">:</span><span class="o">///</span><span class="k">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">docker</span><span class="o">/</span><span class="n">libcontainerd</span><span class="o">/</span><span class="n">docker</span><span class="o">-</span><span class="n">containerd</span><span class="o">.</span><span class="n">sock</span><span class="w"> </span><span class="o">--</span><span class="n">metrics</span><span class="o">-</span><span class="n">interval</span><span class="o">=</span><span class="mi">0</span><span class="w"> </span><span class="o">--</span><span class="n">start</span><span class="o">-</span><span class="n">timeout</span><span class="w"> </span><span class="mi">2</span><span class="n">m</span><span class="w"> </span><span class="o">--</span><span class="n">state</span><span class="o">-</span><span class="n">dir</span><span class="w"> </span><span class="o">/</span><span class="k">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">docker</span><span class="o">/</span><span class="n">libcontainerd</span><span class="o">/</span><span class="n">containerd</span><span class="w"> </span><span class="o">--</span><span class="n">shim</span><span class="w"> </span><span class="n">docker</span><span class="o">-</span><span class="n">containerd</span><span class="o">-</span><span class="n">shim</span><span class="w"> </span><span class="o">--</span><span class="n">runtime</span><span class="w"> </span><span class="n">docker</span><span class="o">-</span><span class="n">runc</span>
<span class="o">...</span>
</code></pre></div>
<p>we now have the API listening to tcp port 2375 but the port is still filtered by our firewall. To configure the firewall add the port to the appropriate zone and interface. Also the range 32000-34000 which is used by jenkins to access the slaves through ssh is needed to be accessible:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># firewall-cmd --permanent --add-port=2375/tcp</span>
<span class="c1"># firewall-cmd --permanent --zone=trusted --change-interface=docker0</span>
<span class="c1"># firewall-cmd --permanent --zone=trusted --add-port=2375/tcp</span>
<span class="c1"># firewall-cmd --permanent --zone=public --add-port=32000-34000/tcp</span>
<span class="c1"># firewall-cmd --reload</span>
<span class="c1"># yum install -y nmap</span>
<span class="c1"># nmap -p 2375 ip.of.docker.host</span>
<span class="n">Starting</span><span class="w"> </span><span class="n">Nmap</span><span class="w"> </span><span class="mf">6.40</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">nmap</span><span class="o">.</span><span class="n">org</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="n">at</span><span class="w"> </span><span class="mi">2017</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">09</span><span class="w"> </span><span class="mi">10</span><span class="p">:</span><span class="mi">29</span><span class="w"> </span><span class="n">CEST</span>
<span class="n">Nmap</span><span class="w"> </span><span class="n">scan</span><span class="w"> </span><span class="n">report</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="mf">10.11</span><span class="o">.</span><span class="mf">1.17</span>
<span class="n">Host</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">up</span><span class="w"> </span><span class="p">(</span><span class="mi">1500</span><span class="n">s</span><span class="w"> </span><span class="n">latency</span><span class="p">)</span><span class="o">.</span>
<span class="n">PORT</span><span class="w"> </span><span class="n">STATE</span><span class="w"> </span><span class="n">SERVICE</span>
<span class="mi">2375</span><span class="o">/</span><span class="n">tcp</span><span class="w"> </span><span class="n">open</span><span class="w"> </span><span class="n">unknown</span>
<span class="n">Nmap</span><span class="w"> </span><span class="n">done</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">IP</span><span class="w"> </span><span class="n">address</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="n">up</span><span class="p">)</span><span class="w"> </span><span class="n">scanned</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="mf">0.05</span><span class="w"> </span><span class="n">seconds</span>
<span class="c1"># curl -X GET http://localhost:2375/images/json</span>
</code></pre></div>
<p>and we are off to go, a docker container can be started using the <a href="https://hub.docker.com/r/jenkins/jenkins/">official upstream image</a> from jenkins. They also did a great job on <a href="https://github.com/jenkinsci/docker/blob/master/README.md">documentation</a> about those docker images. We went for the lts release because this setup will be the production one in the future.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># docker run -d -p 8080:8080 -v /opt/jenkins/:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock jenkinsci/jenkins:lts</span>
</code></pre></div>
<p>as you can see we passed through the <a href="https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/">docker unix socket</a> from the host to the jenkins container by doing so we are now able to instruct actions on the docker host from within our jenkins instance.</p>
<p>on your docker host a jenkins container is running and accessible through the docker host's ip address on port 8080 which is forwarded to the jenkins master docker container. To have access to the container himself or follow the logs:</p>
<div class="highlight"><pre><span></span><code><span class="cp"># docker ps</span>
<span class="n">CONTAINER</span><span class="w"> </span><span class="n">ID</span><span class="w"> </span><span class="n">IMAGE</span><span class="w"> </span><span class="n">COMMAND</span><span class="w"> </span><span class="n">CREATED</span><span class="w"> </span><span class="n">STATUS</span><span class="w"> </span><span class="n">PORTS</span><span class="w"> </span><span class="n">NAMES</span>
<span class="n">d28838216442</span><span class="w"> </span><span class="n">jenkinsci</span><span class="o">/</span><span class="n">jenkins</span><span class="o">:</span><span class="n">lts</span><span class="w"> </span><span class="s">"/bin/tini -- /usr..."</span><span class="w"> </span><span class="mi">19</span><span class="w"> </span><span class="n">hours</span><span class="w"> </span><span class="n">ago</span><span class="w"> </span><span class="n">Up</span><span class="w"> </span><span class="mi">19</span><span class="w"> </span><span class="n">hours</span><span class="w"> </span><span class="mf">0.0.0.0</span><span class="o">:</span><span class="mi">8080</span><span class="o">-></span><span class="mi">8080</span><span class="o">/</span><span class="n">tcp</span><span class="p">,</span><span class="w"> </span><span class="mi">50000</span><span class="o">/</span><span class="n">tcp</span><span class="w"> </span><span class="n">determined_bartik</span>
<span class="cp"># docker exec -i -t --user root d28838216442 /bin/bash</span>
<span class="cp"># docker logs -f d28838216442</span>
</code></pre></div>
<p>the plugin needed to communicate with our docker host is the <a href="https://wiki.jenkins.io/display/JENKINS/Docker+Plugin">docker plugin</a> following the configuration as described in their documentation we created a docker cloud which will be used to execute builds with a specified label only. This is very handy because we can now create different docker images for every piece of software with other dependencies.</p>
<p>For the initial testing setup we used the upstream docker image <a href="https://hub.docker.com/r/evarga/jenkins-slave/">evarga/jenkins-slave</a>.</p>
<h2>docker cloud configuration</h2>
<p>Manage Jenkins -> Configure System: Cloud - Add new cloud: Docker Cloud</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>Name</td>
<td>testdocker</td>
</tr>
<tr>
<td>Docker URL</td>
<td>tcp://ip.address:2375</td>
</tr>
<tr>
<td>Connection Timeout</td>
<td>15</td>
</tr>
<tr>
<td>Read Timeout</td>
<td>5</td>
</tr>
<tr>
<td>Container Cap</td>
<td>100</td>
</tr>
</tbody>
</table>
<p>-> Add Docker Template</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>Docker Image</td>
<td>evarga/jenkins-slave:latest (the tag can be pinpointed for production)</td>
</tr>
<tr>
<td>Container Settings</td>
<td></td>
</tr>
<tr>
<td>Volumes</td>
<td>/var/run/docker.sock:/var/run/docker.sock</td>
</tr>
<tr>
<td>Remote Filing System Root</td>
<td>/home/jenkins</td>
</tr>
<tr>
<td>Labels</td>
<td>generic</td>
</tr>
<tr>
<td>Usage</td>
<td>Only build jobs with label expressions matching this node</td>
</tr>
<tr>
<td>Launch method</td>
<td>Docker SSH computer launcher</td>
</tr>
<tr>
<td>Credentials</td>
<td>jenkins (A dedicated SSH key pair for jenkins use cases)</td>
</tr>
<tr>
<td>Host Key Verification Strategy</td>
<td>Manually provided key Verification Strategy</td>
</tr>
<tr>
<td>SSH-KEY</td>
<td>the private SSH key for the jenkins slave user</td>
</tr>
<tr>
<td>Remote FS Root Mapping</td>
<td>/home/jenkins</td>
</tr>
<tr>
<td>Remove volumes</td>
<td>true</td>
</tr>
<tr>
<td>Pull strategy</td>
<td>Pull once and update latest</td>
</tr>
</tbody>
</table>
<p>Depending on the label specific docker images will be used to execute the job and store the result in a repository.</p>
<p>Along the way I struggled with setting up the docker cloud. In the first place because we are abusing the same docker host as our jenkins container is running on. By passing through the docker socket and opening up a range of ports used by jenkins to SSH into the slaves I finally achieved a running job which spawns a docker container and destroys him as soon as the job is done.</p>
<h2>custom jenkins master image</h2>
<p>to get the docker plugin working nicely the docker daemon needs to be installed on the jenkins instance too. Therefor I created a new <a href="https://hub.docker.com/r/visibilityspots/jenkins-docker/">jenkins-docker</a> docker image which is based on the upstream jenkins:lts image but adds the docker daemon and staticly configure the docker GID to prevent mismatches between the docker socket mounted on the container from the host.</p>
<p>to have the daemon mounted from the host in the jenkins containers, both master as slave should be able to execute docker commands over the socket the docker group need to have the same GID on all containers and 'physical' machines. Since we statically changed this GID already on the containers we also need to map this GID on the physical machine:</p>
<div class="highlight"><pre><span></span><code># groupmod -g 900 docker
# systemctl restart docker.service
</code></pre></div>
<p>The container is started with some extra parameters like the java_opts one for the allocation of RAM memory for the jenkins daemon.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># docker run -d -p 8080:8080 -v /opt/jenkins/:/var/jenkins_home -e JAVA_OPTS="-Xmx6144m" -v /var/run/docker.sock:/var/run/docker.sock visibilityspots/jenkins-docker:latest</span>
</code></pre></div>Docker openstackclient-kilo container2017-05-29T21:00:00+02:002017-08-23T00:00:00+02:00Jantag:visibilityspots.org,2017-05-29:/docker-openstackclient-kilo.html<p>A couple of years ago I deployed an openstack cluster based on <a href="https://www.rdoproject.org/">RDO</a>. Back in the days we implemented the <a href="https://www.openstack.org/software/kilo/">kilo</a> release. Until today we didn't updated yet due to various reasons being no need for the new features, no resources, no time no.. Upgrading would be a better option but we'll have to live with it and since it's running rather well so far we are quite happy with it.</p>
<p>To manage that cloud I use the clients I installed on my local machine, from nova to cinder they all have different packages available for many different platforms. Only …</p><p>A couple of years ago I deployed an openstack cluster based on <a href="https://www.rdoproject.org/">RDO</a>. Back in the days we implemented the <a href="https://www.openstack.org/software/kilo/">kilo</a> release. Until today we didn't updated yet due to various reasons being no need for the new features, no resources, no time no.. Upgrading would be a better option but we'll have to live with it and since it's running rather well so far we are quite happy with it.</p>
<p>To manage that cloud I use the clients I installed on my local machine, from nova to cinder they all have different packages available for many different platforms. Only a couple of weeks ago I noticed the new shiny versions shipping with <a href="https://www.archlinux.org/">Archlinux</a> aren't working anymore with our setup.</p>
<p>So I looked for alternatives, in the short term I logged in on one of the compute nodes or our <a href="https://jenkins.io/">jenkins</a> machine to perform the actions I needed to do. But that's bad practice. So I went a bit further and decided to create a docker container for this. Looking on the docker hub there are already some containers available but they don't specify a specific version.</p>
<p>Also this seemed to me like a great exercise to experience the different stages how a docker container could be built up completely by myself. So I went for it and created a <a href="https://github.com/visibilityspots/dockerfile-openstackclient-kilo">github repository</a> to share my work with the world.</p>
<h2>docker build</h2>
<p>The whole setup relies on one <a href="https://github.com/visibilityspots/dockerfile-openstackclient-kilo/blob/master/Dockerfile">Dockerfile</a>. Since I rely on centos in almost every server environment I'm working on I decided to use the centos:7 <a href="https://hub.docker.com/_/centos/">official</a> docker image.</p>
<p>After some try and error I came to this setup:</p>
<div class="highlight"><pre><span></span><code>FROM centos:7
RUN set -x \
&& yum upgrade -y \
&& yum install -y bash-completion \
&& yum install -y https://repos.fedorapeople.org/repos/openstack/EOL/openstack-kilo/rdo-release-kilo-2.noarch.rpm \
&& yum install -y python-novaclient \
&& yum install -y python-ceilometerclient \
&& yum install -y python-cinderclient \
&& yum install -y python-glanceclient \
&& yum install -y python-heatclient \
&& yum install -y python-ironicclient \
&& yum install -y python-keystoneclient \
&& yum install -y python-manilaclient \
&& yum install -y python-novaclient \
&& yum install -y python-openstackclient \
&& yum install -y python-saharaclient \
&& yum install -y python-troveclient \
&& yum install -y python-zaqarclient \
&& yum clean all \
&& useradd client
USER client
ENTRYPOINT ["bash", "--rcfile", "/home/client/.keystonerc"]
</code></pre></div>
<p>as you could see I opted to install every package using a separate yum install command. That way when a package can't be installed the others aren't infected. Also a yum clean all is performed to cleanup a bit the filesystem to keep the image size a bit under control. By adding a 'client' user I could prevent doing everything as root user which isn't really necessary in our case.</p>
<p>To use the openstack clients, openstack provides a keystonerc file you'll have to source before you could connect through the different API endpoints. I went for the ENTRYPOINT solution which will source a file which can be mounted at run time of the docker container afterwards.</p>
<p>I have to credit my colleague <a href="https://roidelapluie.be/">roidelapluie</a> here since he guided me to this Dockerfile. To be honest about every option :), but hey I learned a lot about the docker world this way!</p>
<p>So right now everyone should be able to build this container himself by checking out the repository and run the docker build command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>git<span class="w"> </span>clone<span class="w"> </span>git@github.com:visibilityspots/dockerfile-openstackclient-kilo.git
$<span class="w"> </span>docker<span class="w"> </span>build<span class="w"> </span>-t<span class="w"> </span>openstackclient-kilo<span class="w"> </span>.
</code></pre></div>
<p>If everything went well the new created docker image should be available on your machine:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>images
REPOSITORY<span class="w"> </span>TAG<span class="w"> </span>IMAGE<span class="w"> </span>ID<span class="w"> </span>CREATED<span class="w"> </span>SIZE
openstackclient-kilo<span class="w"> </span>latest<span class="w"> </span>e89297f75fa8<span class="w"> </span>About<span class="w"> </span>a<span class="w"> </span>minute<span class="w"> </span>ago<span class="w"> </span>331MB
</code></pre></div>
<h2>docker run</h2>
<p>So you could now use the image to spin up an openstack kilo client container:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>--name<span class="w"> </span>openstack-client<span class="w"> </span>--rm<span class="w"> </span>-ti<span class="w"> </span>visibilityspots/openstackclient-kilo
bash-4.2$
</code></pre></div>
<p>As already described before the docker container will automatically source a keystonerc file at the /home/client/.keystonerc path. So you could mount a file from your local machine into the docker container at runtime:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>--name<span class="w"> </span>openstack-client<span class="w"> </span>--rm<span class="w"> </span>-ti<span class="w"> </span>-v<span class="w"> </span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span>/keystonerc_admin:/home/client/.keystonerc<span class="w"> </span>visibilityspots/openstackclient-kilo
<span class="o">[</span>client@a1c38f7635e3<span class="w"> </span>/<span class="o">(</span>keystone_admin<span class="o">)]</span>$<span class="w"> </span>openstack<span class="w"> </span>hypervisor<span class="w"> </span>list
+----+---------------------+
<span class="p">|</span><span class="w"> </span>ID<span class="w"> </span><span class="p">|</span><span class="w"> </span>Hypervisor<span class="w"> </span>Hostname<span class="w"> </span><span class="p">|</span>
+----+---------------------+
<span class="p">|</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>node-01<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>node-02<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span><span class="m">4</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>node-03<span class="w"> </span><span class="p">|</span>
+----+---------------------+
</code></pre></div>
<h2>dgoss (testing)</h2>
<p>Some weeks ago I discovered the docker image testing tool <a href="https://github.com/aelsabbahy/goss/tree/master/extras/dgoss">dgoss</a> which is comparable with <a href="http://serverspec.org/">serverspec</a> to write tests in an easy yaml format to be performed against your docker image.</p>
<p>So I decided to give it a try and wrote a <a href="https://github.com/visibilityspots/dockerfile-openstackclient-kilo/blob/master/goss.yaml">goss.yaml</a> file;</p>
<div class="highlight"><pre><span></span><code><span class="kd">package</span><span class="o">:</span>
<span class="w"> </span><span class="n">rdo</span><span class="o">-</span><span class="n">release</span><span class="o">-</span><span class="n">kilo</span><span class="o">-</span><span class="mi">2</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">novaclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">ceilometerclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">cinderclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">glanceclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">heatclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">ironicclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">keystoneclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">manilaclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">novaclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">openstackclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">saharaclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">troveclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="n">zaqarclient</span><span class="o">:</span>
<span class="w"> </span><span class="n">installed</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="n">file</span><span class="o">:</span>
<span class="w"> </span><span class="sr">/home/client/</span><span class="o">:</span>
<span class="w"> </span><span class="n">exists</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span>
<span class="n">command</span><span class="o">:</span>
<span class="w"> </span><span class="n">openstack</span><span class="w"> </span><span class="o">--</span><span class="n">version</span><span class="o">:</span>
<span class="w"> </span><span class="n">exit</span><span class="o">-</span><span class="n">status</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span>
</code></pre></div>
<p>which will check if all packages are installed correctly and if the commands are working as they should;</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="n">dgoss</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="o">--</span><span class="n">name</span><span class="w"> </span><span class="n">openstack</span><span class="o">-</span><span class="n">client</span><span class="w"> </span><span class="o">--</span><span class="n">rm</span><span class="w"> </span><span class="o">-</span><span class="n">ti</span><span class="w"> </span><span class="o">-</span><span class="n">v</span><span class="w"> </span><span class="err">$</span><span class="p">(</span><span class="n">pwd</span><span class="p">)</span><span class="o">/</span><span class="nl">keystonerc_admin</span><span class="p">:</span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">client</span><span class="o">/</span><span class="p">.</span><span class="n">keystonerc</span><span class="w"> </span><span class="n">visibilityspots</span><span class="o">/</span><span class="n">openstackclient</span><span class="o">-</span><span class="n">kilo</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="n">container</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Container</span><span class="w"> </span><span class="nl">ID</span><span class="p">:</span><span class="w"> </span><span class="mi">539</span><span class="n">f6896</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Sleeping</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="mf">0.2</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Running</span><span class="w"> </span><span class="n">Tests</span>
<span class="k">File</span><span class="err">:</span><span class="w"> </span><span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">client</span><span class="o">/</span><span class="p">.</span><span class="nl">keystonerc</span><span class="p">:</span><span class="w"> </span><span class="ow">exists</span><span class="err">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">cinderclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">ironicclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">manilaclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">rdo</span><span class="o">-</span><span class="k">release</span><span class="o">-</span><span class="n">kilo</span><span class="o">-</span><span class="mi">2</span><span class="err">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">novaclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">saharaclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">zaqarclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">glanceclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">troveclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">heatclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">keystoneclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">ceilometerclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Package</span><span class="p">:</span><span class="w"> </span><span class="n">python</span><span class="o">-</span><span class="nl">openstackclient</span><span class="p">:</span><span class="w"> </span><span class="nl">installed</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">true</span><span class="o">]</span>
<span class="nl">Command</span><span class="p">:</span><span class="w"> </span><span class="n">openstack</span><span class="w"> </span><span class="o">--</span><span class="nl">version</span><span class="p">:</span><span class="w"> </span><span class="k">exit</span><span class="o">-</span><span class="nl">status</span><span class="p">:</span><span class="w"> </span><span class="n">matches</span><span class="w"> </span><span class="nl">expectation</span><span class="p">:</span><span class="w"> </span><span class="o">[</span><span class="n">0</span><span class="o">]</span>
<span class="n">Total</span><span class="w"> </span><span class="nl">Duration</span><span class="p">:</span><span class="w"> </span><span class="mf">0.786</span><span class="n">s</span>
<span class="nf">Count</span><span class="err">:</span><span class="w"> </span><span class="mi">15</span><span class="p">,</span><span class="w"> </span><span class="nl">Failed</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nl">Skipped</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span>
<span class="nl">INFO</span><span class="p">:</span><span class="w"> </span><span class="n">Deleting</span><span class="w"> </span><span class="n">container</span>
</code></pre></div>
<h2>docker hub</h2>
<p>So now we have a working container on our local machine, but since sharing is caring I wanted to upload it to the <a href="https://hub.docker.com/">docker hub</a>. They have a feature to build your docker container based on a github repository. They also have documented the whole setup of <a href="https://docs.docker.com/docker-hub/builds/">automated</a> builds pretty good.</p>
<p>So for every git push a new docker image will be released on the docker hub. But they don't test the image before releasing it. </p>
<h2>travis</h2>
<p>So I looked around and found out this can be achieved by <a href="https://travis-ci.org/">travis</a>. I started writing a <a href="https://github.com/visibilityspots/dockerfile-openstackclient-kilo/blob/master/.travis.yml">travis.yml</a> file which will perform following steps;</p>
<div class="highlight"><pre><span></span><code><span class="n">sudo</span><span class="o">:</span><span class="w"> </span><span class="n">required</span>
<span class="n">services</span><span class="o">:</span>
<span class="o">-</span><span class="w"> </span><span class="n">docker</span>
<span class="n">before_install</span><span class="o">:</span>
<span class="o">-</span><span class="w"> </span><span class="n">curl</span><span class="w"> </span><span class="o">-</span><span class="n">L</span><span class="w"> </span><span class="n">https</span><span class="o">://</span><span class="n">goss</span><span class="o">.</span><span class="na">rocks</span><span class="o">/</span><span class="n">install</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">sudo</span><span class="w"> </span><span class="n">sh</span>
<span class="o">-</span><span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="n">build</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="n">visibilityspots</span><span class="o">/</span><span class="n">openstackclient</span><span class="o">-</span><span class="n">kilo</span><span class="w"> </span><span class="o">.</span>
<span class="n">script</span><span class="o">:</span>
<span class="o">-</span><span class="w"> </span><span class="n">dgoss</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="o">--</span><span class="n">name</span><span class="w"> </span><span class="n">openstack</span><span class="o">-</span><span class="n">client</span><span class="w"> </span><span class="o">-</span><span class="n">ti</span><span class="w"> </span><span class="n">visibilityspots</span><span class="o">/</span><span class="n">openstackclient</span><span class="o">-</span><span class="n">kilo</span>
<span class="n">after_success</span><span class="o">:</span>
<span class="o">-</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"$TRAVIS_BRANCH"</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">"master"</span><span class="w"> </span><span class="o">];</span><span class="w"> </span><span class="n">then</span><span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="n">login</span><span class="w"> </span><span class="o">-</span><span class="n">u</span><span class="o">=</span><span class="n">visibilityspots</span><span class="w"> </span><span class="o">-</span><span class="n">p</span><span class="o">=</span><span class="s2">"$DOCKER_PASSWORD"</span><span class="o">;</span>
<span class="w"> </span><span class="n">docker</span><span class="w"> </span><span class="n">push</span><span class="w"> </span><span class="n">visibilityspots</span><span class="o">/</span><span class="n">openstackclient</span><span class="o">-</span><span class="n">kilo</span><span class="o">;</span><span class="w"> </span><span class="n">fi</span>
<span class="n">env</span><span class="o">:</span>
<span class="w"> </span><span class="n">global</span><span class="o">:</span>
<span class="w"> </span><span class="n">secure</span><span class="o">:</span><span class="w"> </span><span class="n">QA24IWGelQ6nd3Xuwzif8L</span><span class="o">+</span><span class="mi">2</span><span class="n">AeWC6M8V1jkZNJzJxaBLcxpucWcjkmowzaWGPmVfKeFAgfPsgVPfy8gK03JfY3J4H2AcCa8TWhVOoLvvY6ytFLobCWtRI8nIZoYeV8</span><span class="sr">/BNlKzguoE4OEfrloJUU52tQ/NDWVleH5KCBCj3H93eE52USwZ4JUyoZtngd+Zqma0GUomMvOZ0mg2e87UOYQNnCZehh5okAIU34sKGTGwEwWeqxA9xUvBd3l7pRj+5bfeQ07Fn0n3/tmrzFkOKRfCL2HC63Aq0T0LFjKsYza2QykiI5z4enNVjH8d+/05dCCHaj+/ZqKVQWbMi/RIheXk1XvsxPttBlHC03EXdQBfZmiNUUaxtVQ7f9Df0sRPvIrZsYzbHkeqVSHTNIGZLzY2cizLzIedYLOFGKNRM3WsOokIlsn+f/XciZse0D3YPBPzkRlI6sXMLtduxkZLzC1tRgyZhTl1A1kMbXzj94SJzftQ2NW7g0lbSO38DxsZcy4g/VlTDLmDzHvnlcdiU2KPSXgQcPZXQk/AZuZdl/AaYb9FIhP06GYRVRJfxls5qlxIAxxJsBTyBR4UE7SGq86UtZdTqzr/</span><span class="n">LjQXQr8SU</span><span class="o">+</span><span class="n">KfxvKbOvfBiIrqzMcBaRqQgPfTeKhI</span><span class="o">+</span><span class="n">jEhNyAlgnvSQNf8Eg9UgHv6aW51TqyI</span><span class="o">+</span><span class="n">RzVqIs</span><span class="o">=</span>
</code></pre></div>
<p>It will install dgoss and build the container locally on the travis infrastructure. Next it will run the dgoss tests based on the goss.yaml file. When those tests are finished it will upload the created and tested docker container to the docker hub.</p>
<p>To have travis authorized to push to the docker hub we have to provide him our docker credentials in a secure way. Luckily travis thought that through and provided us with the encrypt feature.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>travis<span class="w"> </span>encrypt<span class="w"> </span><span class="nv">DOCKER_PASSWORD</span><span class="o">=</span>YOUR-DOCKER-HUB-PASSWORD<span class="w"> </span>--add
</code></pre></div>
<p>That way the password is stored into the travis environment variable $DOCKER_PASSWORD through an encrypted way as can be seen in your .travis.yaml file.</p>
<p>A detailled log will be provided on <a href="https://travis-ci.org/visibilityspots/dockerfile-openstackclient-kilo">travis</a></p>
<p>To get to this travis.yaml file I combined the <a href="https://docs.travis-ci.com/user/docker/">official</a> travis docker documentation with a <a href="https://sebest.github.io/post/using-travis-ci-to-build-docker-images/">tutorial</a> I found on the net.</p>
<p>Last but not least I had to disable the automatic build feature of the docker hub repository so only the travis builds will be pushed. That can be done on the "Build Settings" of the docker hub repository.</p>Auto deploy webpage using pelican and travis2017-05-21T22:00:00+02:002017-06-01T00:00:00+02:00Jantag:visibilityspots.org,2017-05-21:/pelican-travis.html<p>many years ago I created my own webpage, it all started with pure, HTML evolved to a wordpress and finally became a <a href="https://blog.getpelican.com/">pelican</a> based setup. It got served on many different hosting providers but since a few years it's running on <a href="../aws-migration.html">S3</a> storage and hosted through cloudfront all over the world.</p>
<p>It's a very fast setup, and once the site has been deployed and every little service has been configured and implemented the only thing I need to do is writing content in <a href="https://daringfireball.net/projects/markdown/">markdown</a> without having to consider how to deploy or how it will look.</p>
<p>In this post I'll …</p><p>many years ago I created my own webpage, it all started with pure, HTML evolved to a wordpress and finally became a <a href="https://blog.getpelican.com/">pelican</a> based setup. It got served on many different hosting providers but since a few years it's running on <a href="../aws-migration.html">S3</a> storage and hosted through cloudfront all over the world.</p>
<p>It's a very fast setup, and once the site has been deployed and every little service has been configured and implemented the only thing I need to do is writing content in <a href="https://daringfireball.net/projects/markdown/">markdown</a> without having to consider how to deploy or how it will look.</p>
<p>In this post I'll try to describe how I configured every service, connected them to each other and automated them through <a href="https://travis-ci.org">travis-ci</a>.</p>
<h1>pelican</h1>
<p>it all starts by initiliazing your pelican framework following the <a href="http://docs.getpelican.com/en/3.6.3/quickstart.html">quickstart</a> guide. Before you proceed you can configure and write some initial content for your webpage locally and see how it will look like without having it published to the world.</p>
<p>You can choose a <a href="http://docs.getpelican.com/en/3.6.3/pelican-themes.html">theme</a> of your choose, adding <a href="http://docs.getpelican.com/en/3.6.3/plugins.html">plugins</a> for various use cases or even <a href="http://docs.getpelican.com/en/3.6.3/importer.html">import</a> an existing webpage.</p>
<p>Once you have something you want to publish we can proceed to publish it to the world.</p>
<h1>github</h1>
<p>versioning is something very important in my opinion, by doing so you can easily track changes and collaborate with a team on one web project. Also other people can easily propose changes on your website this way through pull requests. Another purpose of using a github repository is the way we could trigger automation which could deploy our project to different hosting providers.</p>
<p>A nice side effect is that you have a backup in the "cloud".</p>
<p>For the different pelican plugins and themes I use <a href="https://git-scm.com/docs/git-submodule">git submodules</a> so I can easily update them with upstream changes.</p>
<h1>AWS</h1>
<p>as I already mentioned I opted for <a href="https://aws.amazon.com/">AWS</a> to host my blog and some other websites I manage. It's easy to deploy, it's fast and rather cheap compared to other providers, I pay about 30 EUR a year for everything, including domain registration, traffic all over the world and storage.</p>
<h2>IAM</h2>
<p>I learned that using dedicated users for every single use case isn't a bad idea. So for this setup we need a dedicated user with pro-grammatic access, which have only full access to S3 and cloudfront only for the distributions we configure. The generated access and secret keys will be used by travis to upload new content to our S3 bucket and invalidate cache. They can be created by following the <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html">documentation</a></p>
<p>Access for letsencrypt <a href="https://github.com/dlapiduz/certbot-s3front/blob/master/sample-aws-policy.json">policy</a> needs to be granted to the user which will be used to update the blog.</p>
<h2>route53</h2>
<p>Creating your own domain or <a href="http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/MigratingDNS.html">migrating</a> to the DNS service <a href="https://aws.amazon.com/route53/">route53</a> is a very easy way to manage your domain also on amazon. It's easy in the end after all by having one bill for everything.</p>
<p>The only thing I struggled with was the way to update your nameservers after you migrated the domain and made an error in them when migrating. In the route53 configuration pane it can be found in the "Registered domains" tab and not in the hosted zones! Took me some time to figure out the difference between those 2.</p>
<p>Also don't forget to hide your personal data for the different contacts you configured for every registered domain.</p>
<h2>S3</h2>
<p>Amazon <a href="https://aws.amazon.com/s3/">S3</a> object storage service can be used to serve static files and therefore a static <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html">webpage</a> we will be using this feature to host our pelican based website on.</p>
<p>I found a great <a href="http://stackoverflow.com/questions/28675620/cloudfront-redirect-www-to-naked-domain-with-ssl">how to</a> on stackoverflow which explains perfectly how you have to create 2 buckets to redirect between www and the naked domain and how to enable https once you have that feature enabled.</p>
<h2>cloudfront</h2>
<p><a href="https://aws.amazon.com/cloudfront/">cloudfront</a> is amazon's own CDN serving your website around the world on different edge locations. It's easy to <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html">implement</a> with your static S3 based setup too.</p>
<p>Cloudfront caches your site on the different edge locations, by using <a href="https://renzo.lucioni.xyz/s3-deployment-with-travis/">cache invalidation</a> we can trigger the different locations to update their cache according to the new files when being pushed through travis later on.</p>
<h1>Letsencrypt</h1>
<p><a href="https://letsencrypt.org/">letsencrypt</a> is a free, automated and open Certificate Authority which can be used in combination with S3 using the <a href="https://github.com/dlapiduz/certbot-s3front">certbot-s3front</a> tool to get your site served through https.</p>
<p>I automated this process with a script based in /usr/local/bin/:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#</span>
<span class="c1"># renew certificates for X</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">AWS_ACCESS_KEY_ID</span><span class="o">=</span><span class="s2">""</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">AWS_SECRET_ACCESS_KEY</span><span class="o">=</span><span class="s2">""</span>
certbot<span class="w"> </span>--agree-tos<span class="w"> </span>-a<span class="w"> </span>certbot-s3front:auth<span class="w"> </span><span class="se">\</span>
--certbot-s3front:auth-s3-bucket<span class="w"> </span>BUCKET-NAME<span class="w"> </span><span class="se">\</span>
--certbot-s3front:auth-s3-region<span class="w"> </span>REGION<span class="w"> </span><span class="se">\</span>
-i<span class="w"> </span>certbot-s3front:installer<span class="w"> </span><span class="se">\</span>
--certbot-s3front:installer-cf-distribution-id<span class="w"> </span>CLOUDFRONT-ID<span class="w"> </span><span class="se">\</span>
-d<span class="w"> </span>DOMAIN<span class="w"> </span>--renew-by-default<span class="w"> </span>--text
<span class="k">if</span><span class="w"> </span><span class="o">[[</span><span class="w"> </span><span class="nv">$?</span><span class="w"> </span>-ne<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">]]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>/usr/bin/ntfy<span class="w"> </span>-b<span class="w"> </span>telegram<span class="w"> </span>send<span class="w"> </span><span class="s2">"ERROR | Certificate renewal for DOMAIN has failed on </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">!"</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span><span class="p">;</span>
<span class="k">fi</span>
/usr/bin/ntfy<span class="w"> </span>-b<span class="w"> </span>telegram<span class="w"> </span>send<span class="w"> </span><span class="s2">"SUCCESS | Certificates for DOMAIN have been renewed till </span><span class="k">$(</span>date<span class="w"> </span>-d<span class="w"> </span><span class="s2">"3 months"</span><span class="k">)</span><span class="s2"> "</span>
</code></pre></div>
<p>It will source the IAM credentials you created for this specific use case, use certbot to renew your certificate on the specified cloudfront distribution and use <a href="http://ntfy.readthedocs.io/en/latest/">ntfy</a> to inform you about it through telegram in this case.</p>
<p>When the renewal fails it will also sent a notification, I didn't had this feature in the past which lead to expiration of the certificate..</p>
<p>It's triggered by cron:</p>
<div class="highlight"><pre><span></span><code><span class="mf">0</span><span class="w"> </span><span class="mf">0</span><span class="w"> </span><span class="mf">1</span><span class="w"> </span><span class="o">*/</span><span class="mf">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="o">/</span><span class="nb">usr</span><span class="o">/</span><span class="n">local</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="kd">let</span><span class="n">sencrypt</span><span class="o">-*</span>
</code></pre></div>
<p>it will be triggered the <a href="https://crontab.guru/#0_0_1_*/2_*">first day every 2 months</a>, I used this sequence so in case of issues I have time enough to solve it before it expires.</p>
<p>Since we now have everything in place and your website should already be available hosted on AWS we can now automate the whole setup. Meaning the only thing you'll have to perform afterwards is writing content and pushing to git.</p>
<h1>Travis</h1>
<p><a href="https://travis-ci.org/">travis</a> is a tool which enables you to easily write automation tasks every time a new commit has been pushed to your repository. Once you've created your account and linked it to github you'll have to enable travis through their GUI on the repositories you want to monitor for automation.</p>
<p>Once you've done that for your <a href="https://github.com/visibilityspots/blog">repository</a> you'll have to configure some credentials and deploy keys. First you'll need the git deploy key, the process is nicely explained by <a href="https://github.com/steveklabnik/automatically_update_github_pages_with_travis_example">Steve Klabnik</a>. That way you'll have a Github token we'll configure in a bit in our .travis.yaml file.</p>
<p>Besides the github token you'll also need to configure the previously created AWS user access and secret keys in travis so travis will be able to update your S3 bucket and invalidates your caches on cloudfront. You'll need to configure those through the GUI of travis on the particular repository as explained by (renzo)[https://renzo.lucioni.xyz/s3-deployment-with-travis/].</p>
<p>Now the most of the administrative part is done a <a href="https://github.com/visibilityspots/blog/blob/master/.travis.yml">.travis.yaml</a> file is needed in your repository which contains a list of steps to be performed by travis every time a new commit will be pushed.</p>
<h2>dependencies</h2>
<p>The travis file I'm referring to is divided in four parts, the first one being the installation of the different python <a href="https://github.com/visibilityspots/blog/blob/master/requirements.txt">dependencies</a> needed to build and deploy our website.</p>
<div class="highlight"><pre><span></span><code><span class="n">pip</span> <span class="n">install</span> <span class="o">-</span><span class="n">r</span> <span class="n">requirements</span><span class="o">.</span><span class="n">txt</span>
</code></pre></div>
<h2>make</h2>
<p>Secondly we rely on the <a href="https://github.com/visibilityspots/blog/blob/master/Makefile">Makefile</a> to first build and deploy the website to github pages and secondly build it for AWS;</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>make<span class="w"> </span>clean
$<span class="w"> </span>make<span class="w"> </span>github-travis
$<span class="w"> </span>make<span class="w"> </span>clean
$<span class="w"> </span>make<span class="w"> </span>aws-create
</code></pre></div>
<h2>deployment to AWS</h2>
<p>When the website is prepared for AWS it can be deployed using the built in <a href="https://docs.travis-ci.com/user/deployment/s3/">deploy</a> of travis.</p>
<h2>cache invalidation</h2>
<p>Last but not least is the invalidation of the cache on the different cloudfront edge locations so the updated website is also renewed on those servers.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>aws<span class="w"> </span>configure<span class="w"> </span><span class="nb">set</span><span class="w"> </span>preview.cloudfront<span class="w"> </span><span class="nb">true</span>
$<span class="w"> </span>aws<span class="w"> </span>cloudfront<span class="w"> </span>create-invalidation<span class="w"> </span>--distribution-id<span class="w"> </span><span class="nv">$CLOUDFRONT_DISTRIBUTION_ID</span><span class="w"> </span>--paths<span class="w"> </span><span class="s2">"/*"</span>
</code></pre></div>
<p>The result of your build can be followed on the travis webpage as for example the <a href="https://travis-ci.org/visibilityspots/blog/builds/234608263">build</a> of this page</p>Openstack Kilo change MTU till the VM it's tap interface2017-03-23T19:00:00+01:002017-03-23T00:00:00+01:00Jantag:visibilityspots.org,2017-03-23:/openstack-mtu.html<p>Recently I was been asked to increase the MTU on the deployed openstack cluster at one of our customers. Since the beginning of my touch on openstack networking has been the hardest part to get my head around. In the first place because openstack does some nifty things on the networking path. But also cause for the use case at the customer a lot of customization has been done to get it implemented in their infrastructure.</p>
<p>Hence the shiver when the MTU question was been made..</p>
<p>Nevertheless together with a colleague who likes a challenge and has a profound knowledge …</p><p>Recently I was been asked to increase the MTU on the deployed openstack cluster at one of our customers. Since the beginning of my touch on openstack networking has been the hardest part to get my head around. In the first place because openstack does some nifty things on the networking path. But also cause for the use case at the customer a lot of customization has been done to get it implemented in their infrastructure.</p>
<p>Hence the shiver when the MTU question was been made..</p>
<p>Nevertheless together with a colleague who likes a challenge and has a profound knowledge in this area we dived into it. Starting at the external device over all the hardware network switches we came to the openstack cluster, until now nothing got in our way of increasing the MTU size. On the most of the network gear (combination of HP and Cisco) the MTU was already high enough.</p>
<p>But now we came to the compute nodes of our openstack cluster. We have an RDO based kilo release running with one all-in-one controller and a dozen compute nodes. We isolated the compute node where the test instance was running on and went for our dearest friend Mr google for some advise and found a very informational <a href="https://www.openstack.org/assets/presentation-media/the-notorious-mtu.pdf">pdf</a> document about this topic.</p>
<p>After some try and error we got to the current situation where the ovs bridge has been configured with this increased MTU size together with the NIC interface of the compute node itself. This has been achieved by changing following parameters.</p>
<p>To change the MTU size on the ovs bridge we need veth interfaces as described in the configuration file of the plugin.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini</span>
<span class="nv">use_veth_interconnection</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>True
<span class="nv">veth_mtu</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">8000</span>
</code></pre></div>
<p>By restarting the neutron component on the compute node the ovs bridge got reconfigured with veth interfaces and an increased MTU size within seconds and no noticeable down time which was very convenient.</p>
<p>So there is only one step to achieve our goal of sending big packets over the whole chain, the tap interface of the VM on that OVS bridge.</p>
<p>We manually adjusted the tap interfaces by executing an ovs-vsctl command.</p>
<div class="highlight"><pre><span></span><code>ip<span class="w"> </span>link<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mtu<span class="w"> </span><span class="m">8000</span><span class="w"> </span>dev<span class="w"> </span>PORTID
</code></pre></div>
<p>Unfortunately until today we didn't find a fix where a new VM gets a network connection on the OVS bridge with this increased MTU size automatically.</p>
<p>We tried several configurations but no luck.</p>
<p>As a temporary workaround we <a href="http://serverfault.com/questions/680635/mtu-on-open-vswitch-bridge-port">found</a> a loop which reconfigures the MTU sizes for all available ports on an OVS bridge.</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="k">$(</span>ovs-vsctl<span class="w"> </span>list<span class="w"> </span>ports<span class="w"> </span><BRIDGENAME><span class="k">)</span><span class="p">;</span><span class="k">do</span><span class="w"> </span>ip<span class="w"> </span>link<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mtu<span class="w"> </span><span class="m">9000</span><span class="w"> </span>dev<span class="w"> </span><span class="nv">$i</span><span class="p">;</span><span class="k">done</span><span class="p">;</span>ip<span class="w"> </span>a<span class="w"> </span>show<span class="w"> </span><BRIDGENAME>
</code></pre></div>
<p>The unsuccesfull changes;</p>
<div class="highlight"><pre><span></span><code><span class="c1"># /etc/neutron/neutron.conf</span>
<span class="nv">advertise_mtu</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>True
<span class="c1"># /etc/nova/nova.conf</span>
<span class="nv">network_device_mtu</span><span class="o">=</span><span class="m">8000</span>
</code></pre></div>
<p>So if you did found a solution on this part, please en light us!! :)</p>S3stat2016-10-26T19:00:00+02:002016-10-26T00:00:00+02:00Jantag:visibilityspots.org,2016-10-26:/s3stat.html<p>Some weeks ago an article on <a href="https://news.ycombinator.com/item?id=12634447">hacker news</a> got my interest. From time to time I really get an healthy dose of jealousy when people found an idea which could make them buy a tesla. My terms of someone who make a lot of money ;)</p>
<p>This one is so brilliant in it's simplicity that I really was flabbergasted and made me wonder why I never came up with the idea. It generates nice reports of the usage of your site which is hosted by aws. Based on the logs of the S3 bucket or the cloudfront domain you setted up …</p><p>Some weeks ago an article on <a href="https://news.ycombinator.com/item?id=12634447">hacker news</a> got my interest. From time to time I really get an healthy dose of jealousy when people found an idea which could make them buy a tesla. My terms of someone who make a lot of money ;)</p>
<p>This one is so brilliant in it's simplicity that I really was flabbergasted and made me wonder why I never came up with the idea. It generates nice reports of the usage of your site which is hosted by aws. Based on the logs of the S3 bucket or the cloudfront domain you setted up.</p>
<p>As I <a href="../aws-migration.html">blogged</a> about a few months ago I migrated my blog as static content onto an S3 bucket and serve it through the CDN of amazon to the world for a really cheap price. I manage my blog with <a href="http://blog.getpelican.com">pelican</a> which makes a beautiful static website based on markdown files. One of the features is the <a href="http://docs.getpelican.com/en/latest/settings.html?highlight=analytics#themes">google analytics</a> component which sends data through the browser of the visitor. Which can be blocked off course through some inventive add blocking features of the used browser.</p>
<p>So I was trilled when logging creating an account on <a href="https://s3stat.com">s3stat</a> to see what my data is all about in their visual reports. I started by adding my S3 buckets which obviously didn't have logging enabled. I disabled them back in the days when migrating since I didn't saw a use case for them at that moment. This feature can easily be enabled using the separate aws account I created by following their how to guide.</p>
<p>It took a while before the first data got through their website but after a day or two I had some nice and simple reports about the usage of my S3 buckets. One of my blog and one of my custom <a href="https://atlas.hashicorp.com/visibilityspots">vagrant boxes</a>. Besides the delay of about one day I could see what I needed to see.</p>
<p>In comparison with google analytics they offer some more details especially the referral pages are quite interesting to me which is like the only feature I miss in S3stat, they do show your referral pages, but every single page of the website itself is also seen as a referral page. Which kinda creates a lot of pages and it's hard to find the relevant information of external pages.. Google analytics isn't live data neither and since s3stat can't be blocked by some browser plugins they offer more accurate data about your content usage. Another nice feature are the costs, they give you an idea which of the requests is costing you money. Which could be interesting to see if you could adopt your website so it can be cheaper to host it at AWS..</p>
<p>It took some time to get the cloudfront instances coupled, the interface did found the instances but it froze when I selected on of the cloudfront distributions. After a week or two I finally managed to select them and get them coupled through s3stat. My best guess is that the success of the default setting took down some services. I created a support ticket for it but didn't got an answer so far. Guess they are very busy to keep the service up and running.</p>
<p>Since it's working fine now I don't bother about it :) I do have now nice statistics of my blog which is really great and I love it. The only reason I still connect to google analytics are the detailed information about referral websites..</p>
<p>One of the sad parts is the pricing. For my blog it would take about $ 10 each month. Since I only pay $ 2 dollar on average to host it that isn't something I'm willing to pay for those statistics. But I stumbled on their <a href="https://www.s3stat.com/web-stats/cheap-bastard-plan">cheap bastard plan</a> which is the mean reason I wrote a blog post about it.</p>
<p>And because of my empathy with their simplicity :)</p>
<p>I only had trouble the first time I wrote a new article after I setted up s3stat. I'm using <a href="http://s3tools.org/s3cmd">s3cmd</a> to upload new pages to the S3 bucket. And the sync command was deleting the log directory which includes all log entries.. So I had to add the exclude parameter to my s3cmd sync command;</p>
<p><code>--exclude 'log/*'</code></p>
<p>This was a huge mistake from my end. Luckily the files aren't that critical I only lost a few days of them since so nothing to bother about in the end..</p>wireless bond archlinux arm2016-10-17T21:00:00+02:002016-10-17T00:00:00+02:00Jantag:visibilityspots.org,2016-10-17:/wireless-bond-archlinux.html<p>for one of my projects, the <a href="../social-media-wall.html">sms-twitter wall</a> setup, I configured a raspberry pi with 2 wireless network interfaces to connect through a hotspot enabled on an android device. I discovered on previous events that the wireless adapter failed on me from time to time. So I went to the internet to look if I could add a second interface and bond them together.</p>
<p>I found a lot of documentation on how to bond an active-backup strategy with a wired and a wireless interfaces but didn't found a setup with 2 wireless interfaces. After a while I figured out how …</p><p>for one of my projects, the <a href="../social-media-wall.html">sms-twitter wall</a> setup, I configured a raspberry pi with 2 wireless network interfaces to connect through a hotspot enabled on an android device. I discovered on previous events that the wireless adapter failed on me from time to time. So I went to the internet to look if I could add a second interface and bond them together.</p>
<p>I found a lot of documentation on how to bond an active-backup strategy with a wired and a wireless interfaces but didn't found a setup with 2 wireless interfaces. After a while I figured out how it can be accomplished, even with a static ip. This static ip was necessary because the phone is receiving text messages converting them and sending them to the pi's ip address so the smswall could handle them.</p>
<p>Another side note was the wired connection, when at home I wanted to plug the ethernet cable so I could connect to my NAS for backups and to by pass the slow mobile connection for debugging purposes.</p>
<p>A lot of usable documentation I found on the archwiki obviously, like the <a href="https://wiki.archlinux.org/index.php/netctl#Bonding">netctl article</a></p>
<p>A range of tools needs to be installed on the system, which I did using <a href="https://archlinux.fr/yaourt-en">yaourt</a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>yaourt<span class="w"> </span>-S<span class="w"> </span>netctl<span class="w"> </span>wpa_supplicant<span class="w"> </span>ifenslave
</code></pre></div>
<p>The bonding kernel module needs to be installed and configured</p>
<p>vim /etc/modules-load.d/bonding.conf</p>
<div class="highlight"><pre><span></span><code>bonding
/etc/modprobe.d/bonding.conf
</code></pre></div>
<p>options bonding mode=active-backup miimon=100 primary=wlan0 max_bonds=0</p>
<p>the netctl wired connection profile
/etc/netcltl/wired</p>
<div class="highlight"><pre><span></span><code>Description='A basic static ethernet connection'
Interface=eth0
Connection=ethernet
IP=static
Address=('192.168.0.106/24')
Routes=('default metric 1 via 192.168.0.1')
DNS=('192.168.0.1')
ExcludeAuto=no
Priority=2
</code></pre></div>
<p>the netctl profile failover needs to be configured also with a static ip</p>
<p>/etc/netctl/failover</p>
<div class="highlight"><pre><span></span><code>Description="bond interface"
Interface=bond0
Connection=bond
BindsToInterfaces=(wlan0 wlan1)
IP=static
Address=('192.168.43.150/24')
Routes=('default metric 100 via 192.168.43.1')
DNS=('192.168.43.1')
</code></pre></div>
<p>notice the Router parameter instead of the Gateway, by assigning different metrics both profiles can be used together. Else the will be fighting with each other to configure the default route. And in my experience obviously the one you need (wired) will loose over and over again..</p>
<p>Enable the profiles</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>start<span class="w"> </span>netctl-ifplugd@eth0.service
$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>status<span class="w"> </span>netctl-ifplugd@eth0.service
$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>netctl-ifplugd@eth0.service
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>netctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>failover
</code></pre></div>
<p>If you like me already had the wireless interface configured in the netctl-auto modus be sure to disable and stop that service!</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>status<span class="w"> </span>netctl-ifplugd@eth0.service
$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>stop<span class="w"> </span>netctl-ifplugd@eth0.service
$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>disable<span class="w"> </span>netctl-ifplugd@eth0.service
</code></pre></div>
<p>We should now configure the wireless authentication for both wireless adapters. First calculate the <a href="https://wiki.archlinux.org/index.php/WPA_supplicant#Connecting_with_wpa_passphrase">wpa_passphrase</a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>wpa_passphrase<span class="w"> </span>SSID<span class="w"> </span>passphrase
</code></pre></div>
<p>and copy over the output into the following configuration files</p>
<p>/etc/wpa_supplicant/wpa_supplicant-wlan0.conf</p>
<div class="highlight"><pre><span></span><code>ctrl_interface=/run/wpa_supplicant-wlan0
update_config=1
network={
ssid="SSID"
#psk="passphrase"
psk=XX
}
</code></pre></div>
<p>/etc/wpa_supplicant/wpa_supplicant-wlan1.conf</p>
<div class="highlight"><pre><span></span><code>ctrl_interface=/run/wpa_supplicant-wlan1
update_config=1
network={
ssid="SSID"
#psk="passphrase"
psk=XX
}
</code></pre></div>
<p>and their dependency configuration files</p>
<p>/etc/systemd/system/wpa_supplicant@wlan0.service.d/customdependency.conf</p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span>
<span class="na">After</span><span class="o">=</span><span class="s">netctl@failover.service</span>
</code></pre></div>
<p>/etc/systemd/system/wpa_supplicant@wlan1.service.d/customdependency.conf</p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span>
<span class="na">After</span><span class="o">=</span><span class="s">wpa_supplicant@wlan0.service</span>
</code></pre></div>
<p>I had to configure them in this order since they didn't came up properly if I didn't only one could achieve to authenticate instead of both. By having the wlan1 unit depend on the wlan0 I had tackled it.</p>
<p>When rebooting the device you should see them both connected using 'iwconfig' and only the bond0 and eth0 interfaces having an ip address using 'ifconfig' or 'ip -4 a'</p>
<p>to see the bond status</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="nx">cat</span><span class="w"> </span><span class="o">/</span><span class="nx">proc</span><span class="o">/</span><span class="nx">net</span><span class="o">/</span><span class="nx">bonding</span><span class="o">/</span><span class="nx">bond0</span>
<span class="nx">Ethernet</span><span class="w"> </span><span class="nx">Channel</span><span class="w"> </span><span class="nx">Bonding</span><span class="w"> </span><span class="nx">Driver</span><span class="p">:</span><span class="w"> </span><span class="nx">v3</span><span class="m m-Double">.7.1</span><span class="w"> </span><span class="p">(</span><span class="nx">April</span><span class="w"> </span><span class="mi">27</span><span class="p">,</span><span class="w"> </span><span class="mi">2011</span><span class="p">)</span>
<span class="nx">Bonding</span><span class="w"> </span><span class="nx">Mode</span><span class="p">:</span><span class="w"> </span><span class="nx">fault</span><span class="o">-</span><span class="nx">tolerance</span><span class="w"> </span><span class="p">(</span><span class="nx">active</span><span class="o">-</span><span class="nx">backup</span><span class="p">)</span>
<span class="nx">Primary</span><span class="w"> </span><span class="nx">Slave</span><span class="p">:</span><span class="w"> </span><span class="nx">wlan0</span><span class="w"> </span><span class="p">(</span><span class="nx">primary_reselect</span><span class="w"> </span><span class="nx">always</span><span class="p">)</span>
<span class="nx">Currently</span><span class="w"> </span><span class="nx">Active</span><span class="w"> </span><span class="nx">Slave</span><span class="p">:</span><span class="w"> </span><span class="nx">wlan0</span>
<span class="nx">MII</span><span class="w"> </span><span class="nx">Status</span><span class="p">:</span><span class="w"> </span><span class="nx">up</span>
<span class="nx">MII</span><span class="w"> </span><span class="nx">Polling</span><span class="w"> </span><span class="nx">Interval</span><span class="w"> </span><span class="p">(</span><span class="nx">ms</span><span class="p">):</span><span class="w"> </span><span class="mi">100</span>
<span class="nx">Up</span><span class="w"> </span><span class="nx">Delay</span><span class="w"> </span><span class="p">(</span><span class="nx">ms</span><span class="p">):</span><span class="w"> </span><span class="mi">0</span>
<span class="nx">Down</span><span class="w"> </span><span class="nx">Delay</span><span class="w"> </span><span class="p">(</span><span class="nx">ms</span><span class="p">):</span><span class="w"> </span><span class="mi">0</span>
<span class="nx">Slave</span><span class="w"> </span><span class="nx">Interface</span><span class="p">:</span><span class="w"> </span><span class="nx">wlan0</span>
<span class="nx">MII</span><span class="w"> </span><span class="nx">Status</span><span class="p">:</span><span class="w"> </span><span class="nx">up</span>
<span class="nx">Speed</span><span class="p">:</span><span class="w"> </span><span class="nx">Unknown</span>
<span class="nx">Duplex</span><span class="p">:</span><span class="w"> </span><span class="nx">Unknown</span>
<span class="nx">Link</span><span class="w"> </span><span class="nx">Failure</span><span class="w"> </span><span class="nx">Count</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span>
<span class="nx">Permanent</span><span class="w"> </span><span class="nx">HW</span><span class="w"> </span><span class="kd">addr</span><span class="p">:</span><span class="w"> </span><span class="err">##</span><span class="p">:</span><span class="err">##</span><span class="w"> </span>
<span class="nx">Slave</span><span class="w"> </span><span class="nx">queue</span><span class="w"> </span><span class="nx">ID</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span>
<span class="nx">Slave</span><span class="w"> </span><span class="nx">Interface</span><span class="p">:</span><span class="w"> </span><span class="nx">wlan1</span>
<span class="nx">MII</span><span class="w"> </span><span class="nx">Status</span><span class="p">:</span><span class="w"> </span><span class="nx">up</span>
<span class="nx">Speed</span><span class="p">:</span><span class="w"> </span><span class="nx">Unknown</span>
<span class="nx">Duplex</span><span class="p">:</span><span class="w"> </span><span class="nx">Unknown</span>
<span class="nx">Link</span><span class="w"> </span><span class="nx">Failure</span><span class="w"> </span><span class="nx">Count</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span>
<span class="nx">Permanent</span><span class="w"> </span><span class="nx">HW</span><span class="w"> </span><span class="kd">addr</span><span class="p">:</span><span class="w"> </span><span class="err">##</span><span class="p">:</span><span class="err">##</span>
<span class="nx">Slave</span><span class="w"> </span><span class="nx">queue</span><span class="w"> </span><span class="nx">ID</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span>
</code></pre></div>
<p>I did some testing by unplugging interfaces, bringing them down through the cli and my system kept online. Even without the ethernet cable plugged ;)</p>
<p>So after some debugging I once again won a fight over the network!! </p>AWS migration2016-09-18T21:00:00+02:002017-03-17T00:00:00+01:00Jantag:visibilityspots.org,2016-09-18:/aws-migration.html<p>About a year ago I attended the <a href="http://aws.amazon.com/events/awsome-day/benelux/belgium/">AWSome Day</a> at <a href="http://lamot-mechelen.be">Mechelen</a>. Back then I wrote a first draft article about it, but it got out of my sight unfortunately. I reviewed it and decided to publish it anyway.</p>
<p>The event was based on their essentials course and took use through the different AWS core services (compute, storage, database and network).</p>
<p>I do know it has nothing to see with open-source. But it is a part of that ultimate cloud based setup I believe in which exists in one central place from where you can manage all your virtual machines independent …</p><p>About a year ago I attended the <a href="http://aws.amazon.com/events/awsome-day/benelux/belgium/">AWSome Day</a> at <a href="http://lamot-mechelen.be">Mechelen</a>. Back then I wrote a first draft article about it, but it got out of my sight unfortunately. I reviewed it and decided to publish it anyway.</p>
<p>The event was based on their essentials course and took use through the different AWS core services (compute, storage, database and network).</p>
<p>I do know it has nothing to see with open-source. But it is a part of that ultimate cloud based setup I believe in which exists in one central place from where you can manage all your virtual machines independent of which stack/service it uses.</p>
<p>In that ultimate setup the public cloud is important to me too. And when you look at public clouds, amazon can't just be ignored in my opinion. That should give you the flexibility to extend your infrastructure when needed, add the ability to benchmark applications between different virtualization / storage resources and make managing them easier without having to open up way too many management consoles.</p>
<h1>The course</h1>
<p>the course itself was intended for both technical as management background profiles. The goal of it was merely to highlight the different products and what you can technically achieve with them as well as how they could be combined.</p>
<h2>introduction</h2>
<p>Starting with an introduction on the different AWS services and the console going over the many different options, unfortunately the live demos where postponed since the speaker forgot his charger.</p>
<h2>storage</h2>
<p>After the introduction the different storage services provided by AWS where enlighted, focusing on the <a href="http://aws.amazon.com/s3/details/">S3</a> and <a href="http://aws.amazon.com/ebs/details/">EBS</a> instances. Where clearly told the first one is an object storage system and the second one can be used to deploy filesystems on it.</p>
<h2>console demo</h2>
<p>The speaker also pulled my attention by mentioning you could serve a static website on an S3 instance. Since you only buy for what you use this has been the trigger for me to start looking into migrating my current blog to an amazon hosted one. Which I'll describe further on in this post.</p>
<p>Once he got his charger back the speaker showed us the aws management console and the different options and features you could use. During the demo he also pointed to the <a href="http://aws.amazon.com/whitepapers/aws-security-best-practices/">security best practices</a> like having MFA enabled, not using your root account and such..</p>
<p>So I went buy myself an <a href="http://onlinenoram.gemalto.com/">ezio display card</a> to enhance my geeky nerd state.</p>
<h2>compute services and networking</h2>
<p>Next topic of the day concerned the different services of computing instances and networking. Starting with the <a href="http://aws.amazon.com/ec2/details/">EC2</a> by explaining their different tastes and flavors depending on what you want to achieve.</p>
<p>Also a tip of the speaker was to benchmark the application you want to provision on different types of instances. It only costs you the amount of time running the benchmark tests. But in the end you should have found the right instance for what you are trying to achieve at a reasonable price in the long term!</p>
<h2>databases</h2>
<p>two types of databases are briefly touched first the <a href="https://aws.amazon.com/rds/">RDS</a> one which is the more classic alternative amazon provides.</p>
<p>And the <a href="https://aws.amazon.com/dynamodb/">dynamodb</a> which is like the nosql database managed cloud service.</p>
<p>And when this all is what you were looking for you could start perhaps your own amazon based <a href="http://aws.amazon.com/vpc/details/">Virtual Private Cloud</a></p>
<h1>open guides</h1>
<p>I stumbled onto <a href="https://github.com/open-guides/og-aws">open guides</a> for aws which is a collection of very useful information and references for every aws service combined with how-to guides and such. Very useful when playing around with the AWS services and need information about one of them!</p>
<h1>dynamic DNS</h1>
<p>It is even possible to get rid of all the free dyndns services you are using and use the route53 API to update your DNS names for certain appliances. By following the guide of <a href="https://willwarren.com/2014/07/03/roll-dynamic-dns-service-using-amazon-route53/">Will Warren</a> I now have my own domain name used for dynamic DNS names.</p>
<h1>Static blog</h1>
<p>During the presentation the speaker mentioned you could host a static website on an amazon S3 instance. A couple of years ago I migrated my blog to a <a href="http://getpelican.com">pelican</a> based one. This tool allows you to write your articles in markdown and convert those into a static html based instance. Since I don't need interactivity for my blog it's the perfect solution to me. The only aspect I need to take into account is to write the actual content. So I don't have to focus about the other stuff like layout and such.</p>
<p>I used to host my website on a traditional hosting service called <a href="http://one.com">one.com</a>. Costs me about 30 EUR a year for some web space and a domain name. Since it felt like a great exercise to get to know the different AWS services I decided to host my blog on those technologies as a proof of concept to start with.</p>
<p>I followed the <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html">tutorial</a> from the aws documentation to get it up and running.</p>
<p>Using the cloudfront functionality the content of my static blog from the s3 instance will be populated through the different edge locations of the <a href="http://aws.amazon.com/cloudfront/details">CloudFront Global Edge Network</a>. By linking my new domain name visibilityspots.org which is a <a href="http://aws.amazon.com/route53/details/">Route 53</a> instance to the cloudfront instance, the end users requests are automatically routed to the nearest edge location for high performance delivery of your content.</p>
<p>This makes my blog super fast on any continent for a rather cheap price!</p>
<p>I once read an article about the https encryption on the internet. The author believed in a world where only encrypted web traffic should exist so nobody has to care about anymore if their data is encrypted on the web. My blog doesn't has any use case where I really need this encryption. But once again it's a great exercise to set it up to get a feeling how this stuff actually works.</p>
<p>So I went for a <a href="https://www.startssl.com/">start ssl</a> domain ssl certificate which costs me nothing but a monthly reactivation mail. This certificate I uploaded to my aws account, as described by <a href="https://bryce.fisher-fleig.org/blog/setting-up-ssl-on-aws-cloudfront-and-s3">Bryce Fisher</a> so I could start using it to serve my blog with the world through an encrypted line.</p>
<p>In the meantime <a href="https://letsencrypt.org/">letsencrypt</a> was founded and I switched my start ssl certificate to a letsencrypt one. Using the <a href="https://github.com/dlapiduz/letsencrypt-s3front">letsencrypt-s3front</a> tool from <a href="https://github.com/dlapiduz">Diego Lapiduz</a> this got really easy, and I even got it automated through my <a href="../raspberry-pi.html">pi</a> so every x months the certificate is renewed and I get a notification through <a href="http://ntfy.readthedocs.io/en/latest/">ntfy</a> on telegram about it as soon as it's done.</p>
<h2>Price</h2>
<p>I have my blog served by AWS about more then a year now and it costs me about $ 1.5 every month. With the annual fee of $ 12 for the domain name it costs me about the same as before at one.com. Only now my blog is supersonic fast and available through amazons cloudfront service.</p>
<h2>Benchmarks</h2>
<p>I did a test using the <a href="https://www.joedog.org/siege-home/">siege</a> software on 4 different platforms where I do host my blog. The one.com hosting which is serving the visibilityspots.com domain, the github pages one, the s3 instance directly and the cloudfront cached visibilityspots.org domain.</p>
<p>I did expected the one.com domain would be ending at the bottom of the performance tests. Which did indeed turns out as I thought. Rather unexpected was the platform of github scoring the highest on the test. I could have directed my DNS to the github pages but over the time I experienced some down time of github from time to time. It's not that I could save a lot of money by using github so I decided to keep AWS cloudfront as my primary hosting partner.</p>
<p>I decided to execute the benchmarks once again after more than one year and as you can see the results are a lot better compared to a year ago. And my decision to stick with AWS has payed of as you can see they are a lot faster than github those days.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># siege -b -t5M http://visibilityspots.github.io/blog/</span>
Date<span class="w"> </span><span class="p">&</span><span class="w"> </span>Time,<span class="w"> </span>Trans,<span class="w"> </span>Elap<span class="w"> </span>Time,<span class="w"> </span>Data<span class="w"> </span>Trans,<span class="w"> </span>Resp<span class="w"> </span>T,<span class="w"> </span>Trans<span class="w"> </span>R,<span class="w"> </span>Thrghput,<span class="w"> </span>Concur,<span class="w"> </span>OKAY,<span class="w"> </span>Fail
<span class="m">2015</span>-06-03<span class="w"> </span><span class="m">22</span>:35:02,<span class="w"> </span><span class="m">2794</span>,<span class="w"> </span><span class="m">299</span>.46,<span class="w"> </span><span class="m">47</span>,<span class="w"> </span><span class="m">0</span>.11,<span class="w"> </span><span class="m">9</span>.33,<span class="w"> </span><span class="m">0</span>.16,<span class="w"> </span><span class="m">1</span>.00,<span class="w"> </span><span class="m">2794</span>,<span class="w"> </span><span class="m">0</span>
<span class="m">2016</span>-09-18<span class="w"> </span><span class="m">19</span>:31:23,<span class="w"> </span><span class="m">14293</span>,<span class="w"> </span><span class="m">299</span>.94,<span class="w"> </span><span class="m">24</span>,<span class="w"> </span><span class="m">0</span>.41,<span class="w"> </span><span class="m">47</span>.65,<span class="w"> </span><span class="m">0</span>.08,<span class="w"> </span><span class="m">19</span>.39,<span class="w"> </span><span class="m">14293</span>,<span class="w"> </span><span class="m">1</span>
<span class="c1"># siege -b -t5M http://visibilityspots.org</span>
<span class="m">2015</span>-06-03<span class="w"> </span><span class="m">22</span>:40:45,<span class="w"> </span><span class="m">2642</span>,<span class="w"> </span><span class="m">299</span>.14,<span class="w"> </span><span class="m">44</span>,<span class="w"> </span><span class="m">0</span>.11,<span class="w"> </span><span class="m">8</span>.83,<span class="w"> </span><span class="m">0</span>.15,<span class="w"> </span><span class="m">1</span>.00,<span class="w"> </span><span class="m">2642</span>,<span class="w"> </span><span class="m">0</span>
<span class="m">2016</span>-09-18<span class="w"> </span><span class="m">19</span>:38:46,<span class="w"> </span><span class="m">16613</span>,<span class="w"> </span><span class="m">299</span>.90,<span class="w"> </span><span class="m">79</span>,<span class="w"> </span><span class="m">0</span>.44,<span class="w"> </span><span class="m">55</span>.40,<span class="w"> </span><span class="m">0</span>.26,<span class="w"> </span><span class="m">24</span>.44,<span class="w"> </span><span class="m">16614</span>,<span class="w"> </span><span class="m">0</span>
<span class="c1"># siege -b -t5M http://visibilityspots.org.s3-website-eu-west-1.amazonaws.com/</span>
<span class="m">2015</span>-06-03<span class="w"> </span><span class="m">22</span>:52:32,<span class="w"> </span><span class="m">1686</span>,<span class="w"> </span><span class="m">299</span>.03,<span class="w"> </span><span class="m">28</span>,<span class="w"> </span><span class="m">0</span>.18,<span class="w"> </span><span class="m">5</span>.64,<span class="w"> </span><span class="m">0</span>.09,<span class="w"> </span><span class="m">1</span>.00,<span class="w"> </span><span class="m">1686</span>,<span class="w"> </span><span class="m">0</span>
<span class="m">2016</span>-09-18<span class="w"> </span><span class="m">19</span>:45:07,<span class="w"> </span><span class="m">14063</span>,<span class="w"> </span><span class="m">299</span>.40,<span class="w"> </span><span class="m">73</span>,<span class="w"> </span><span class="m">0</span>.53,<span class="w"> </span><span class="m">46</span>.97,<span class="w"> </span><span class="m">0</span>.24,<span class="w"> </span><span class="m">24</span>.66,<span class="w"> </span><span class="m">14063</span>,<span class="w"> </span><span class="m">0</span>
<span class="c1"># siege -b -t5M http://visibilityspots.com</span>
<span class="m">2015</span>-06-03<span class="w"> </span><span class="m">22</span>:27:47,<span class="w"> </span><span class="m">1617</span>,<span class="w"> </span><span class="m">299</span>.56,<span class="w"> </span><span class="m">27</span>,<span class="w"> </span><span class="m">0</span>.19,<span class="w"> </span><span class="m">5</span>.40,<span class="w"> </span><span class="m">0</span>.09,<span class="w"> </span><span class="m">1</span>.00,<span class="w"> </span><span class="m">1617</span>,<span class="w"> </span><span class="m">0</span>
</code></pre></div>
<p>Since the results of all those actions where rather satisfying I decided to migrate all the services I had at one.com to AWS. By using a second domain I could play around without interrupting my existing services. Once each one of them was running I redirected the DNS records to the .org ones.</p>
<p>A static blog on S3, a mail service, <a href="http://www.openwebanalytics.com/">OWA</a> instance, my little roomba project and an <a href="https://owncloud.org">owncloud</a> S3 storage were running on amazon. I still had a VPS running in the field serving an owncloud instance which I used for my calendars (caldav) and address books (carddav) syncing with my laptop <a href="https://vdirsyncer.pimutils.org/en/stable/">vdirsyncer</a> and my android phone <a href="https://davdroid.bitfire.at/">davdroid</a> and sharing pictures with the family.</p>
<p>Since the performance of the photo page was rather unsatisfying, and storage became an issue I bought myself a Synology <a href="https://www.synology.com/en-us/support/download/DS214play">DS214play</a> NAS so I migrated all my services to my own little cloud storage running at home. Right now only my blog is served on AWS and all other services are running on my NAS. I don't rely on the public cloud anymore for any of my services.</p>
<p>Only an offsite backup of some of my encrypted data containers is synced once in a while through <a href="https://aws.amazon.com/glacier/">glacier</a>. Which is only used in case of geological disaster happens and both my parents and parents-in-law computes devices are destroyed together with my own (which are in sync using <a href="https://syncthing.net/">syncthing</a> and my off site backup disk at work got destroyed. Which really sounds paranoia now I write about it :)</p>
<p>I moved all this because of some privacy matters I have regarding all the public cloud services provides.</p>
<p>And until today that combination really worked out very well. I don't loose a lot of time in maintaining the different platforms but I can focus on using and configuring new services like home automation using home-assistant, a wemos sensor framework, and many more. </p>
<p>Which I will blog about in the future.</p>Openstack static ip2016-09-05T19:00:00+02:002016-09-05T00:00:00+02:00Jantag:visibilityspots.org,2016-09-05:/openstack-static-ip.html<p>last couple of days I have been fighting with the way an static ip is configured on an openstack virtual centos 6 instance. In our specific use case we ditched as many network openstack services as possible as I <a href="https://visibilityspots.org/openstack-layer2.html">previously</a> described.</p>
<p>We want to have the instances running in our current network spaces of the R&D department. In this department until some days ago we didn't had any DHCP server running. But a few weeks back we added an extra remote network space into our platform where we configured a remote compute-node.</p>
<p>This is where the issues started popping …</p><p>last couple of days I have been fighting with the way an static ip is configured on an openstack virtual centos 6 instance. In our specific use case we ditched as many network openstack services as possible as I <a href="https://visibilityspots.org/openstack-layer2.html">previously</a> described.</p>
<p>We want to have the instances running in our current network spaces of the R&D department. In this department until some days ago we didn't had any DHCP server running. But a few weeks back we added an extra remote network space into our platform where we configured a remote compute-node.</p>
<p>This is where the issues started popping up.</p>
<p>In openstack when you spin up an instance with a fixed ip, it will basicly create a neutron port for it, attach it to the vm's NIC interface who will get the ip through DHCP. This makes sense since you want to have your basic images as abstract as possible. But since we disabled a lot of the neutron/openstack network logic our vm got an ip address served by an external dhcp service. Which obviously wasn't what we where looking for.</p>
<p>So we digged in the documentation of cloud-init, openstack and network interfaces. Not much has been documented, or at least we couldn't find it easily about the metadata served by a so called configuration drive.</p>
<p>I figured the metadata is attached through either a /dev/vdb disk or a /dev/sr0 cd-rom drive. In the ec2 metadata the local-ip is indicating the static ip address assigned to the created port. After some looking around the possibilities we decided to write a little script which will fetch this information, rewrite the interface configuration and restart the network service to get the static up and running.</p>
<p>The script is located in /usr/local/bin/reconfigure-static-ip-eth0</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="k">if</span><span class="w"> </span><span class="k">$(</span>lsblk<span class="w"> </span>-l<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-q<span class="w"> </span>sr0<span class="k">)</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>mount<span class="w"> </span>/dev/sr0<span class="w"> </span>/mnt
<span class="k">elif</span><span class="w"> </span><span class="k">$(</span>lsblk<span class="w"> </span>-l<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-q<span class="w"> </span>vdb<span class="k">)</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>mount<span class="w"> </span>/dev/vdb<span class="w"> </span>/mnt
<span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Mountpoint of metadata not found"</span>
<span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">1</span><span class="p">;</span>
<span class="k">fi</span>
splitip<span class="w"> </span><span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="nb">local</span><span class="w"> </span>IFS
<span class="w"> </span><span class="nv">IFS</span><span class="o">=</span>.
<span class="w"> </span><span class="nb">set</span><span class="w"> </span>--<span class="w"> </span><span class="nv">$*</span>
<span class="w"> </span><span class="nv">GATEWAY</span><span class="o">=</span><span class="nv">$GATEWAY</span><span class="s2">"."</span><span class="nv">$@</span>
<span class="o">}</span>
<span class="nv">STATIC_IP</span><span class="o">=</span><span class="k">$(</span>grep<span class="w"> </span>-ri<span class="w"> </span>local-ipv4<span class="w"> </span>/mnt/<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span><span class="s1">','</span><span class="w"> </span><span class="s1">'\n'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>local-ip<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span>-F<span class="w"> </span><span class="s1">' '</span><span class="w"> </span><span class="s1">'{print $2}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="k">)</span>
<span class="nv">GATEWAY</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$STATIC_IP</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>cut<span class="w"> </span>-d<span class="s2">"."</span><span class="w"> </span>-f1-3<span class="sb">`</span><span class="s2">".1"</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"VLAN=no"</span><span class="w"> </span>><span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
<span class="nb">echo</span><span class="w"> </span><span class="s2">"NOZEROCONF=yes"</span><span class="w"> </span>>><span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
<span class="nb">echo</span><span class="w"> </span><span class="s2">"NETMASK=255.255.255.0"</span><span class="w"> </span>>><span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
<span class="nb">echo</span><span class="w"> </span><span class="s2">"BOOTPROTO=static"</span><span class="w"> </span>>><span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
<span class="nb">echo</span><span class="w"> </span><span class="s2">"USERCTL=no"</span><span class="w"> </span>>><span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
<span class="nb">echo</span><span class="w"> </span><span class="s2">"IPADDR=</span><span class="nv">$STATIC_IP</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
<span class="nb">echo</span><span class="w"> </span><span class="s2">"ONBOOT=yes"</span><span class="w"> </span>>><span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
<span class="nb">echo</span><span class="w"> </span><span class="s2">"GATEWAY=</span><span class="nv">$GATEWAY</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
<span class="nb">echo</span><span class="w"> </span><span class="s2">"DEVICE=eth0"</span><span class="w"> </span>>><span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
cat<span class="w"> </span>/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/init.d/network<span class="w"> </span>restart
umount<span class="w"> </span>/mnt
</code></pre></div>
<p>By calling this script through rc.local (/etc/rc.local) it will reconfigure the network interface right after all services are started. In our use case the instance is only used as a hop through different separated network environments so no services are relying on the network interface.</p>
<p>During our little search we couldn't believe we are the only ones hitting against this issue, I do hope others will read this post and comment with more clean ways to do so but this did solved it for us in a rather clean and fast way to go further with the development of the actual products behind those hop through nodes.</p>Openstack live-migration2016-04-21T19:00:00+02:002016-04-21T00:00:00+02:00Jantag:visibilityspots.org,2016-04-21:/openstack-live-migration.html<p>Some of you may already have notices others just stumbled on this post through a search engine, I have set up an openstack private cloud at one of our projects:</p>
<ul>
<li><a href="https://visibilityspots.org/vlan-flat-neutron-provider.html">vlan flat-neutron provider network</a></li>
<li><a href="https://visibilityspots.org/openstack-layer2.html">layer2</a></li>
</ul>
<p>We have noticed that the benefits of having a private cloud is spreading through the different teams within the organization and therefore the interest into this flexibility is growing. Since this wasn't the original <a href="https://visibilityspots.org/vlan-flat-neutron-provider.html">use case</a> we are encountering some design issues right now.</p>
<p>For the original instances the default <a href="http://docs.openstack.org/openstack-ops/content/compute_nodes.html#overcommit">overcommit</a> ratios are fine. But the request for new machines with other goals are like …</p><p>Some of you may already have notices others just stumbled on this post through a search engine, I have set up an openstack private cloud at one of our projects:</p>
<ul>
<li><a href="https://visibilityspots.org/vlan-flat-neutron-provider.html">vlan flat-neutron provider network</a></li>
<li><a href="https://visibilityspots.org/openstack-layer2.html">layer2</a></li>
</ul>
<p>We have noticed that the benefits of having a private cloud is spreading through the different teams within the organization and therefore the interest into this flexibility is growing. Since this wasn't the original <a href="https://visibilityspots.org/vlan-flat-neutron-provider.html">use case</a> we are encountering some design issues right now.</p>
<p>For the original instances the default <a href="http://docs.openstack.org/openstack-ops/content/compute_nodes.html#overcommit">overcommit</a> ratios are fine. But the request for new machines with other goals are like interfering with those original instances running in the same default <a href="http://docs.openstack.org/openstack-ops/content/scaling.html#az_s3">availability zone</a>.</p>
<p>So we are looking to configure some <a href="http://docs.openstack.org/openstack-ops/content/scaling.html#ha_s3">aggregate zones</a> to keep this under control. As soon as we figure out a workable solution I will write about it in a new blog post.</p>
<p>But in the discussions to come to a solution one remark was , couldn't openstack tackle the issues of having an hypervisor with a growing load and memory issues itself by migrating instances to another hypervisors? Which is like a valuable argument to me. So before even looking into such a solution the feature of live migration should work..</p>
<p>Since we aren't using shared storage for our cloud this could be tricky. So I went to the web to inform myself about the different options.</p>
<p>I came across some very interesting reads, like the one of <a href="https://thornelabs.net/2014/06/14/do-not-use-shared-storage-for-openstack-instances.html">Thornelabs</a> why you shouldn't use shared storage for example. Which has some valuable disadvantages of it besides the benefits. In our use case the benefits aren't outweighing against disadvantages. But as I have noticed in the whole openstack story there are options for almost every cloud use case and therefore the logical complexity of it. So for many amongst you shared storage could be a solution.</p>
<p>Another rather interesting one about live migration as a <a href="https://www.blueboxcloud.com/insight/blog-article/live-migration-is-a-perk-not-a-panacea">perk not a panacea</a></p>
<p>One of the <a href="http://docs.openstack.org/openstack-ops/content/compute_nodes.html#instance_storage">options</a> is <a href="http://docs.openstack.org/openstack-ops/content/compute_nodes.html#on_compute_node_storage_nonshared">non shared</a> storage, the default of the RDO packstack installer, based on LVM. On our setup we are using this default.</p>
<p>This has the consequence we can only use the live migration about with the kvm block storage <a href="http://www.sebastien-han.fr/blog/2012/07/12/openstack-block-migration/">migration</a> which isn't really <a href="http://osdir.com/ml/openstack-cloud-computing/2012-08/msg00293.html">supported</a> by the upstream developers and will probably phased out in the future for something more reliable.</p>
<p>We configured <a href="http://docs.openstack.org/user-guide/cli_config_drive.html">config drives</a> as the default to get the metadata served to cloud-init at boot time for an instance. The default drive format (iso9660) has a bug in libvirt of copying a read-only disk. To tackle this one we configured the vfat format on all hypervisors.</p>
<p>Unfortunately this still doesn't solve our issue with it. Apparently when you use the live migrate option openstack <a href="https://bugs.launchpad.net/nova/+bug/1214943">doesn't</a> take the overcommit ratio into account. Since our cloud is already overcommitted we don't have enough resources according to the live migration precheck to move instances around..</p>
<p>The proposed fix isn't released yet in the RDO kilo nova packages and patching a system isn't something I like to do in a semi-production environment.</p>
<p>So until today live migration isn't something we have tackled yet on our cloud. If you have solved this on your kilo RDO release cloud already feel free to enlighten me about it!</p>Openstack layer22016-02-26T20:30:00+01:002016-04-21T00:00:00+02:00Jantag:visibilityspots.org,2016-02-26:/openstack-layer2.html<p>A few months ago I implemented an RDO based openstack kilo release private cloud at one of our customers for their development platform. Through time we tackled a couple of issues so the cloud could be fitted into their work flows.</p>
<p>We stumbled onto some minor issues and some major ones. Let's begin with the minor ones ;)</p>
<p>When upgrading the all-in-one controller before we started using the cloud in 'production' a mean <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1284978">bug</a> bit us in the ankle due to a new hiera package. After some digging around a <a href="https://review.openstack.org/#/c/249301/3/packstack/modules/ospluginutils.py">patch</a> came to the rescue together with the exclusion of the …</p><p>A few months ago I implemented an RDO based openstack kilo release private cloud at one of our customers for their development platform. Through time we tackled a couple of issues so the cloud could be fitted into their work flows.</p>
<p>We stumbled onto some minor issues and some major ones. Let's begin with the minor ones ;)</p>
<p>When upgrading the all-in-one controller before we started using the cloud in 'production' a mean <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1284978">bug</a> bit us in the ankle due to a new hiera package. After some digging around a <a href="https://review.openstack.org/#/c/249301/3/packstack/modules/ospluginutils.py">patch</a> came to the rescue together with the exclusion of the packages puppet<em> and hiera</em> from the epel repository</p>
<div class="highlight"><pre><span></span><code>vim /etc/yum.repos.d/epel.repo +9
exclude=hiera*,puppet
</code></pre></div>
<p>Once launched some rather irritating behavior seemed to be the default timeout of horizon (openstack-dashboard) of 30 minutes. It's a development cloud after all and people didn't wanted to re login all the time. To change the default timeout we added a SESSION_TIMEOUT parameter to the local_settings file and restarted apache</p>
<div class="highlight"><pre><span></span><code>vim /etc/openstack-dashboard/local_settings
SESSION_TIMEOUT = 28800
systemctl restart httpd
</code></pre></div>
<p>After which we reconfigured the expiration time of the keystone token and restarted the keystone service</p>
<div class="highlight"><pre><span></span><code>vim /etc/keystone/keystone.conf
expiration = 28800
systemctl restart openstack-keystone.service
</code></pre></div>
<p>Another rather tiny issue was the access to the console through the web interface. It couldn't connect through the instance when running on another compute node.</p>
<p>After some research it seemed to by DNS. In the development setup no DNS has been configured for the different compute nodes. We tackled it by the proxy client setting in nova.conf on every compute node</p>
<div class="highlight"><pre><span></span><code><span class="nx">vim</span><span class="w"> </span><span class="o">/</span><span class="nx">etc</span><span class="o">/</span><span class="nx">nova</span><span class="o">/</span><span class="nx">nova</span><span class="o">/</span><span class="nx">conf</span>
<span class="nx">vncserver_proxyclient_address</span><span class="p">=</span><span class="nx">actual</span><span class="p">.</span><span class="nx">ip</span><span class="p">.</span><span class="nx">of</span><span class="p">.</span><span class="nx">the</span><span class="p">.</span><span class="nx">server</span>
</code></pre></div>
<p>During maintenance we had to reboot a compute-node, after this reboot the instances living on the compute node where not started and came up in shutdown state.</p>
<p>It seemed to be the default behavior of openstack, but as always there is a configuration parameter for it to force the instances to start after a reboot.</p>
<div class="highlight"><pre><span></span><code>vim /etc/nova/nova.conf
resume_guests_state_on_host_boot=true
</code></pre></div>
<p>The default behavior of the kilo RDO deployed cloud to pass data through cloud-init to the instance is by serving the file through a separate network at boot time.</p>
<p>We however found it more efficient to serve this file through the filesystem, the so called <a href="http://docs.openstack.org/user-guide/cli_config_drive.html">config_drive</a> option.</p>
<p>It can be forced through the nova config file</p>
<div class="highlight"><pre><span></span><code>vim /etc/nova/nova.conf
force_config_drive=True
</code></pre></div>
<p>Also be aware with the lock_passwd feature with passing users through cloud init in openstack due to a <a href="https://bugs.launchpad.net/cloud-init/+bug/1521554">bug</a></p>
<p>So now the minor issues are tackled let's switch to the major and more impacting ones.</p>
<p>We digged into the behavior of the network during the proof of concept phase. And hell that's a big challenge! By default openstack neutron creates a linux bridge and an open vswitch bridge.</p>
<p>The linux bridge is used to translate the security groups into iptables configured to the linux bridge interfaces.</p>
<p>In our use case we wanted to keep the networking setup as simple as possible. Mainly cause a lot of the instance will be used for testing purposes including network traffic and performance.</p>
<p>Since it's only used for internal usage into the R&D department on the we decided to disable the security groups and therefore to ditch the linux bridge out of the linux cluster.</p>
<p>Another reason to disable the security groups was the fact that by using the nova interface-attach command of a network without a subnet configured (layer2) only the default security group was applied to this extra interface.</p>
<p>I filled out a <a href="https://bugs.launchpad.net/neutron/+bug/1512645">bug</a> for this, but it seems we are abusing a bug (attaching networks without subnet configured to an instance) as a feature.</p>
<p>Openstack really doesn't like you to take over control of the layer3 networking part after all. But in our use case we really needed to take over this control to keep our instances as abstract and dynamic as possible.</p>
<p>We only need layer2 connectivity when adding an instance to a specific VLAN based network, an ip address is provided by another DHCP server on this VLAN network. So we don't want openstack to provide DHCP addresses.</p>
<p>By using the nova interface-attach command and the bug/feature of having networks without subnets configured attached we achieved to meet this goal of layer2 connectivity.</p>
<p>So to disable the linux bridge you'll need to <a href="https://gist.github.com/djoreilly/db9c2d32a473c6643551">disable</a> the security groups</p>
<div class="highlight"><pre><span></span><code>vim /etc/nova/nova.conf
security_group_api = nova
firewall_driver = nova.virt.firewall.NoopFirewallDriver
vim /etc/neutron/plugins/ml2/ml2_conf.ini # (only on controller node)
enable_security_group = False
vim /etc/neutron/plugins/openvswitch/
firewall_driver = neutron.agent.firewall.NoopFirewallDriver
</code></pre></div>
<p>Last but not least some configuration parameters got changed through without we noticed probably by some misconfigured settings in <a href="https://wiki.openstack.org/wiki/Packstack">packstack</a>. To keep this under control we made the configuration files immutable so they could only be modified by manual changes.</p>
<div class="highlight"><pre><span></span><code>chattr +i /etc/neutron/plugins/ml2/ml2_conf.ini # (only on controller node)
chattr +i /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
chattr +i /etc/nova/nova.conf
chattr +i /etc/neutron/policy.json
</code></pre></div>
<p>Those are about the main issues worth mentioning we went through. By sharing those I hope to help others in their quest to tame the openstack cluster. If any question arise feel free to comment or contact me about them!</p>SMS twitter wall2015-11-03T23:00:00+01:002016-10-17T00:00:00+02:00Jantag:visibilityspots.org,2015-11-03:/social-media-wall.html<p>A long time ago I was active in the local scouting group. To earn some money to keep our group in a healthy financial position we organized a so called vedettweekend. It's an event where people can have a beer, <a href="http://vedett.be">vedett</a> obviously spin a dance, enjoy a spaghetti and have a good chat with your friends.</p>
<p>I always played with the idea of creating a social media wall on one of the projectors we rented for the event. But I never found/made time for it, until today.. My youngest brother is who's interests are overlapping with mine brought up …</p><p>A long time ago I was active in the local scouting group. To earn some money to keep our group in a healthy financial position we organized a so called vedettweekend. It's an event where people can have a beer, <a href="http://vedett.be">vedett</a> obviously spin a dance, enjoy a spaghetti and have a good chat with your friends.</p>
<p>I always played with the idea of creating a social media wall on one of the projectors we rented for the event. But I never found/made time for it, until today.. My youngest brother is who's interests are overlapping with mine brought up the idea again. Since he's already managing the <a href="https://drupal.org">drupal</a> in combination with the facebook page of the organization he was thinking of combining that knowledge to set up a social media page and serve it to one of the projectors using a raspberry pi.</p>
<p>So I picked up back the idea and started looking around on the internet to a neat open-source solution which could help us achieve our goal. I found a solution by <a href="http://sms-wall.org">sms-wall</a> an open-source piece of software which shows tweets and sms messages in real time.</p>
<p>A disadvantage of this software is the language although the documentation is very detailed it's hard to translate it from time to time from french to english/dutch. Same does count for the text displayed through the web application, I created a <a href="https://github.com/assobug/smswall/pull/2">pull request</a> to enhance more languages for the front end of the application so this application could reach more people around the globe!</p>
<p>But honestly, this is the only disadvantage, I succeeded setting this up in less then a day and spent more time in setting up the underlying components.</p>
<h1>hardware</h1>
<p><img alt="overview" src="../../images/smswall/overview.jpg"></p>
<ul>
<li>HTC <a href="http://www.htc.com/be-nl/support/htc-one-v/">One V</a> running <a href="http://cyanogenmod.org">cyanogenmod</a></li>
<li><a href="https://www.raspberrypi.org/products/raspberry-pi-2-model-b/">Raspberry pi</a> B+ running <a href="http://archlinuxarm.org/">alarm</a></li>
<li>xtorm XPD06 - <a href="http://www.xtorm.eu/en/power-hubs/smart-hub/">smart hub</a></li>
<li>bluetooth keyboard <a href="http://www.amazon.com/Rii-RT-MWK02-Wireless-Keyboard-Pointer/dp/B00BVZZG0C/ref=cm_cr_pr_product_top?ie=UTF8">RT-MWK02</a></li>
</ul>
<p>In a first reaction I was going to configure the sms-wall on a VPS or laptop and using the pi to show the stream to the public. But when thinking about the whole idea it just triggered my interest if I could configure it all together on a pi and therefore having a mobile setup which can be deployed everywhere.</p>
<h1>software</h1>
<h2>operating system</h2>
<p>As an archlinux addict off course I will try to get it working on this lightweight distribution. Which seems like the perfect use case for a raspberry pi. Archlinux doesn't come with a lot of services running which you will never gonna need. Every single piece of software you need you have to install and configure yourself. That way only the needed services are occupying resources! Which is exactly what I want in like all my use cases :)</p>
<p>So I started by installing archlinux to the pi as described in the <a href="http://archlinuxarm.org/platforms/armv7/broadcom/raspberry-pi-2">installation guide</a>, once this base setup was running on both the wired as the wireless network I could ssh into the raspberry to start the installation of the different services needed for the operationality of smswall.</p>
<p>It's one of the topics on my todo list to configure a raspberry pi so I could use him as an emergency work station. That means my current window and display manager on a archlinux distribution on a pi. Since this project also needs an output I started to configure this pi with the <a href="https://wiki.archlinux.org/index.php/SLiM">slim</a> display manager and <a href="https://wiki.archlinux.org/index.php/Ratpoison">ratpoison</a> window manager.</p>
<p>To get them working you also need <a href="https://wiki.archlinux.org/index.php/Xorg">xorg</a> and the xf86-video-fbdev driver.</p>
<p>I configured slim to auto login and used .xinitrc in the users home dir to start ratpoison</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="k">exec</span><span class="w"> </span><span class="nv">ratpoison</span>
</code></pre></div>
<p>That way when booting up the pi my window manager is started up immediately.</p>
<h2>services</h2>
<p>Smswall does need a web service and a database, so I installed and configured <a href="https://wiki.archlinux.org/index.php/Apache_HTTP_Server">apache</a>, <a href="https://wiki.archlinux.org/index.php/MySQL">mariadb</a> and glued them all together through the different configuration options. Don't forget to install and configure the different <a href="https://wiki.archlinux.org/index.php/Apache_HTTP_Server#Extensions">extensions</a> for apache too, like php and the pdo database drivers.</p>
<p>Once those dependencies are installed I got myself an headache by following the french <a href="https://github.com/assobug/smswall/tree/master/smswall#smswall">documentation</a> to install the actual service.</p>
<p>I also followed their guidelines to use the python grabber instead of the php tweet grabber and configured an old android phone running cyanogenmod through tasker so it delivers sms messages immediately to the pi itself.</p>
<h3>wireless</h3>
<p>To accomplish this behavior I configured the phone as a hotspot and staticly assigned an ip address to the pi. Using this static pi the phone could deliver the incoming messages through the API on served on the static ip on the pi.</p>
<p>Through this hotspot the raspberry also has connectivity to the outside world using mobile data.</p>
<p>During previous events I experienced some malfunction on the wireless adapter, so I decided to introduce bonding which I spent an entire <a href="../wireless-bond-archlinux.html">blog post</a> about.</p>
<h3>reverse tunnel</h3>
<p>As port forwarding isn't an option on belgian mobile networks I configured a reverse ssh tunnel to an upstream VPS. A systemctl script will run in the background and try when not already up and having an internet connection available to bring up this tunnel.</p>
<p>By doing so I can access the pi from everywhere I have internet connectivity and through some port forwardings I could even access the smswall and admin page myself.</p>
<h3>pagekite</h3>
<p>When the pi is also connected to a local LAN through it's ethernet port the reverse tunnel is working fine. But for some events that uplink isn't an option. And for some reason the reverse tunnel option didn't worked out fine.</p>
<p>So I looked around on the net and stumbled onto <a href="https://www.pagekite.org">pagekite</a>. After setting up a custom frontend on a VPS I could connect the kite running on the raspberry to it. That way I didn't had to subscribe to the free service of pagekite itself and keeping control of the tunnel.</p>
<p>It worked fine through the cli but took me a while to have it started on the pi at boottime. Added it to the startup script;</p>
<h3>startup script</h3>
<p>I also wrote a little startup script which waits till there is internet connectivity, start the pagekite service, starts the tweet grabber and the <a href="https://wiki.archlinux.org/index.php/Luakit">luakit</a> web browser which points directly to the smswall.</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="k">while</span><span class="w"> </span>!<span class="w"> </span>ping<span class="w"> </span>-c1<span class="w"> </span>www.google.com<span class="w"> </span><span class="p">&</span>>/dev/null<span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span>:<span class="p">;</span><span class="w"> </span><span class="k">done</span>
/bin/bash<span class="w"> </span>-c<span class="w"> </span><span class="s2">"pagekite &"</span>
<span class="nb">cd</span><span class="w"> </span>/srv/http/smswall/
/bin/bash<span class="w"> </span>-c<span class="w"> </span><span class="s2">"source env/bin/activate; cd stream; python grabber.py &"</span>
xset<span class="w"> </span>s<span class="w"> </span>off
xset<span class="w"> </span>-dpms
luakit<span class="w"> </span>http://localhost/smswall/smswall
</code></pre></div>
<h1>result</h1>
<p>at home I don't have a beamer to test this setup so I attached the pi to our television screen.</p>
<p><img alt="screen" src="../../images/smswall/screen.jpg"></p>
<p>And I have to say it really looked nice already on this rather little screen. My first idea was to hook up the pi to a powerbank and hook the powerbank to the electricity as a cheap UPS. Unfortunately all powerbanks I tried weren't able to handle both at the same time.</p>
<p>So I directly connected the pi to the electricity instead. Found also my old carkit for the phone and attached him to the raspberry pi case.</p>
<p><img alt="carkit" src="../../images/smswall/carkit.jpg"></p>
<p>During the event the first evening from about 23 hrs till 4 in the morning about 900 messages were sent through SMS. Tweets where rather rare, about 10 tweets where sent using the hashtag #vedettweekend.</p>
<p>As murphy came through the second day the wifi dongle gave up, since debugging on that big beamer screen wasn't really an option since the event was already started I decided to break up the wall that second evening.</p>
<h2>lessons learned</h2>
<p>I need some more robust way of communication between the pi and the raspberry or by bonding 2 wifi dongles or by using the USB tethering instead of wifi.</p>
<p>I spent some money on a tiny tft lcd screen to attach directly to the pi. As soon as it arrives I will try to configure it in a way I could use it for debugging when no hdmi cable is connected to the pi.</p>
<p>It was a real blast and worked perfectly the first night, still we have to warn you, no moderation or filtering has been configured on sms wall. Some people really will abuse this and send some nasty shit to your wall!</p>
<p>On the other hand you do have their phone numbers right now up to you what to do with those ;)</p>
<h1>future</h1>
<p>I will try to enhance his setup for the next event, maybe with facebook integration and other social networks. Or with a special mobile number so for every message a part can be used to donate.</p>
<p>I'll update this post with those enhancements and so keep you posted about the progress on this one.</p>Vagrant puppet setup2015-10-10T23:00:00+02:002015-10-10T00:00:00+02:00Jantag:visibilityspots.org,2015-10-10:/vagrant-puppet-setup.html<p>We at <a href="https://inuits.eu">Inuits</a> are using vagrant for a lot of use cases, neither you are a developer or a sysadmin you for sure will walk into it. Me, myself I do use it merely to automate the many different use cases asked by various projects. It took some time to get myself organized with this pretty nifty piece of software.</p>
<p>In the beginning I used it with the default virtualization provider <a href="https://virtualbox.org">virtualbox</a> later on I switched to <a href="https://visibilityspots.org/vagrant-setup.html">lxc</a> containers instead. By using those containers I already gained on performance. Spinning up and down new containers to test if an application …</p><p>We at <a href="https://inuits.eu">Inuits</a> are using vagrant for a lot of use cases, neither you are a developer or a sysadmin you for sure will walk into it. Me, myself I do use it merely to automate the many different use cases asked by various projects. It took some time to get myself organized with this pretty nifty piece of software.</p>
<p>In the beginning I used it with the default virtualization provider <a href="https://virtualbox.org">virtualbox</a> later on I switched to <a href="https://visibilityspots.org/vagrant-setup.html">lxc</a> containers instead. By using those containers I already gained on performance. Spinning up and down new containers to test if an application is deployed fully automatically got 2 times as fast as when using vm's.</p>
<p>But what I struggled with the most where the many different projects. Each time a new piece of software needed to be automated I copied over the puppet base I used the previous time. Which lead to outdated setups for older projects, many duplicate code over and over again. When updating base modules for both the puppet agent as the puppetmaster previous projects got forgotten..</p>
<p>So I tried to figure out a way I could keep them all up to date with the same code base. We are using <a href="https://git.org">git</a> for almost all our projects as our versioning platform. So I figured out I could use the features of git to achieve the goals I've setted for my setup. One code base I could update without interrupting the functionality of the different proof of concepts but with the availability of upgrading those in a easy way.</p>
<p>So I started my <a href="https://github.com/visibilityspots/vagrant-puppet.git">vagrant-puppet</a> project on github. In the master branch a base setup has been configured with a puppetmaster container and a client container. Both are running the latest stable release of puppet 3.x. The puppetmaster is setted up using puppetdb puppetserver or apache/passenger it can be used both with the <a href="https://atlas.hashicorp.com/visibilityspots/boxes/centos-6.x-puppet-3.x">centos6</a> or <a href="https://atlas.hashicorp.com/visibilityspots/boxes/centos-7.x-puppet-3.x">centos7</a> containers I crafted using the <a href="https://github.com/visibilityspots/vagrant-lxc-base-boxes">lxc-base-boxes</a> repository.</p>
<h1>puppet</h1>
<p>to automate the different pieces of software we do use puppet, if upstream puppet-modules are available I pull those in through git submodules if not I write my own.</p>
<p>By using the vagrant rsync functionality I could write my module and hiera data in my own preferred environment since they are synced to the running puppetmaster through rsync.</p>
<p>This syncronisation can be achieved in two ways. Manually:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vagrant<span class="w"> </span>rsync
</code></pre></div>
<p>Or setting up a daemon:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vagrant<span class="w"> </span>rsync-auto
</code></pre></div>
<p>That way changes you made through your local environment are synced into the puppetmaster container.</p>
<h1>upgrade</h1>
<h2>submodules - upstream puppet modules</h2>
<p>Every once in a while when a new version of puppet has been released I try to keep my container vagrant boxes up to date. Once those are upgraded I get myself into to master branch and update all the used git submodules with this one liner:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>git<span class="w"> </span>submodule<span class="w"> </span>foreach<span class="w"> </span>git<span class="w"> </span>pull<span class="w"> </span>origin<span class="w"> </span>master
</code></pre></div>
<p>This way the latest released version of the different used upstream puppet module repositories are fetched into the master branch.</p>
<p>And try to provision my puppetmaster from scratch, depending on the changes been done in the different puppet modules I need to adopt my <a href="https://github.com/visibilityspots/vagrant-puppet/tree/master/hieradata">hieradata</a>. By looking into the hieradata you could see I'm using the puppet roles and profiles principle. With one simple trick in the <a href="https://github.com/visibilityspots/vagrant-puppet/blob/master/puppet/environments/production/manifests/site.pp">site.pp</a> pointed out by <a href="https://twitter.com/PeetersSimon">one</a> of my colleagues I created a role based hierarchy in my hieradata. Based on the role fact given in the node hiera data the parameters needed to get the functionality of the role configured are fetched from the role's hieradata.</p>
<p>By doing so the hiera data of a particular role can be easily reused without having to keep them in sync on every node who needs the same role.</p>
<p>I still need to figure out a way I can achieve the same behavior based on profiles data.</p>
<p>This way my master branch keeps staying in sync with the latest releases on the different puppet tools.</p>
<h2>different projects, different branches</h2>
<p>But I wanted to go a step further, by getting all my different projects in one place to ease the maintainability of them. So I started tinkering about it. The first idea consisted of having them all in one environment like it would be the case in the real world.</p>
<p>But this has a big disadvantage. It would be a mess in the future when a lot of such proof of concepts are combined in one puppet environment. Also an unneeded level of complexity would been added if you want to show of one particular project to a customer or an interested fellow through the interweb.</p>
<p>It looked for me this was the perfect use case for branches. Every branch created from the master branch already got a working puppetmaster client setup and can easily be upgraded by merging the master branch into it when upgrades are released.</p>
<p>By checking out a branch a subset of different submodules can be loaded after the previous ones are cleaned up:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>git<span class="w"> </span>checkout<span class="w"> </span>feature_branch
$<span class="w"> </span>git<span class="w"> </span>clean<span class="w"> </span>-d<span class="w"> </span>-f<span class="w"> </span>-f
$<span class="w"> </span>git<span class="w"> </span>submodule<span class="w"> </span>update<span class="w"> </span>--init<span class="w"> </span>--recursive
</code></pre></div>
<p>This way the specific puppet modules for a specific projects are loaded with a known working version. When merging from the upgraded master branch the submodules are updated to.</p>
<h2>merging master branch</h2>
<p>when the master branch has been upgraded I now can easily merge those updates into the different feature branches:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>git<span class="w"> </span>checkout<span class="w"> </span>feature_branch
$<span class="w"> </span>git<span class="w"> </span>clean<span class="w"> </span>-d<span class="w"> </span>-f<span class="w"> </span>-f
$<span class="w"> </span>git<span class="w"> </span>merge<span class="w"> </span>origin/master
$<span class="w"> </span>git<span class="w"> </span>submodule<span class="w"> </span>update<span class="w"> </span>--init<span class="w"> </span>--recursive
$<span class="w"> </span>vagrant<span class="w"> </span>up<span class="w"> </span>puppetmaster<span class="w"> </span>--provider<span class="o">=</span>lxc
$<span class="w"> </span>vagrant<span class="w"> </span>up<span class="w"> </span>node<span class="w"> </span>--provider<span class="o">=</span>lxc
</code></pre></div>
<p>By configuring this setup I know have a flexible environment to test deploy and write new puppet code when some piece of software needs to be automated on my local machine with a puppetmaster almost simultaneous to a production one.</p>Openstack vlan based flat neutron network provider2015-09-29T19:00:00+02:002015-10-02T00:00:00+02:00Jantag:visibilityspots.org,2015-09-29:/vlan-flat-neutron-provider.html<p>at one of my projects I was been asked to set up a private cloud for a validation platform. The whole idea behind this proof of concept is based on the flexibility to spin up and down certain instances providing some specific functionality so tests banks can be ran against them.</p>
<p>As soon as the tests are finished the machines could be terminated. Those instances should be configured using some configuration management software, like <a href="https://puppetlabs.com/puppet/puppet-open-source">puppet</a>. That way the instances are rebuildable and could be treated as cattle.</p>
<p>On the other hand, it takes about 20 minutes to build up an …</p><p>at one of my projects I was been asked to set up a private cloud for a validation platform. The whole idea behind this proof of concept is based on the flexibility to spin up and down certain instances providing some specific functionality so tests banks can be ran against them.</p>
<p>As soon as the tests are finished the machines could be terminated. Those instances should be configured using some configuration management software, like <a href="https://puppetlabs.com/puppet/puppet-open-source">puppet</a>. That way the instances are rebuildable and could be treated as cattle.</p>
<p>On the other hand, it takes about 20 minutes to build up an instance from scratch, centos minimal with a puppet run to install and configure the whole needed stack. So we looked for a workable way to spin up instances really quick without the waiting time of 20 minutes every time.</p>
<p>We found a workable solution with <a href="https://packer.io">packer</a>. By configuring a template which describes a series of steps needed to be executed to get a fully working instance based on a centos minimal cloud instance, we could provide an easy and reusable way to build our artifacts.</p>
<p>When running the packer command an openstack instance is launched based on a <a href="http://cloud.centos.org/centos/">centos cloud</a> image. Packer will use rsync to upload some needed data directories, in our case a puppet environment. Once this step has been done a local puppet apply will be performed based on the previously uploaded puppet environment. As soon as this puppet run has been successfully executed an image will be created an immediately be uploaded to your openstack instance.</p>
<p>By using <a href="https://vagrantup.com">vagrant</a> you could easily write your puppet code first and test it against a local vm based on <a href="https://virtualbox.org">virtualbox</a> or <a href="https://github.com/fgrehm/vagrant-lxc">lxc</a> containers. Once you know your puppet manifests are working on a local vm you could test it on an openstack instance using the <a href="https://github.com/ggiamarchi/vagrant-openstack-provider">vagrant-openstack</a> provider. That way you could filter out some unforeseen issues without the need of running packer over and over again.</p>
<p>When your vagrant-openstack based instance is deployed fine packer is used to build an image of your specific device.</p>
<p>By spinning up an instance based on this crafted image you could gain like about 18 minutes every time you launch one since it takes about less than 2 minutes to get it up and running fully functional!</p>
<h1>Openstack</h1>
<p>We used the RDO <a href="https://www.rdoproject.org/Quickstart">all-in-one</a> installer to get an openstack up and running on one physical machine rather quickly (15-30 minutes for the initial services).</p>
<p>To set openstack up without the demo data:</p>
<div class="highlight"><pre><span></span><code># packstack --allinone --provision-demo=n
</code></pre></div>
<p>This openstack instance is based on <a href="https://www.centos.org/download/">CentOS 7 minimal</a> since it's a requirement of the used openstack release <a href="https://wiki.openstack.org/wiki/ReleaseNotes/Kilo">kilo</a>.</p>
<h2>networking</h2>
<p>in our case we wanted some different networking setup as from the default one with natting. Instead we wanted a <a href="https://trickycloud.wordpress.com/2013/11/12/setting-up-a-flat-network-with-neutron/">flat</a> network provider so our instances have an ip within the same range as our development network. That way the natting could be kicked out of the setup to exclude some possible networking performance.</p>
<p>Beside this flat network we do use vlan's too, so the openstack instance should be able to route over those vlan's too. We found a similar setup on the <a href="http://www.s3it.uzh.ch/blog/openstack-neutron-vlan/">blog</a> of the University of Zurich. But it lacked an underlying physical network configuration example on the all-in-one node itself.</p>
<p>On opencloudblog a clear <a href="http://www.opencloudblog.com/?p=460">article</a> helped in trying to understand the network philosophy used to get it working.</p>
<h3>manual network configuration</h3>
<p>creating the vlan bridge which is used by openstack to communicate with the physical vlan based networking switch:</p>
<div class="highlight"><pre><span></span><code>ovs-vsctl<span class="w"> </span>add-br<span class="w"> </span>br-vlan
ovs-vsctl<span class="w"> </span>add-port<span class="w"> </span>br-vlan<span class="w"> </span>eth0
vconfig<span class="w"> </span>add<span class="w"> </span>br-vlan<span class="w"> </span><span class="m">100</span>
</code></pre></div>
<p>configuring an ip from the development range to an interface on the node so we have access to it:</p>
<div class="highlight"><pre><span></span><code>ip<span class="w"> </span>link<span class="w"> </span><span class="nb">set</span><span class="w"> </span>br-vlan<span class="w"> </span>up
ip<span class="w"> </span>link<span class="w"> </span><span class="nb">set</span><span class="w"> </span>br-vlan.100<span class="w"> </span>up
ip<span class="w"> </span>address<span class="w"> </span>add<span class="w"> </span>dev<span class="w"> </span>br-vlan.100<span class="w"> </span><span class="m">192</span>.168.0.100<span class="w"> </span>netmask<span class="w"> </span><span class="m">255</span>.255.255.0
ip<span class="w"> </span>route<span class="w"> </span>add<span class="w"> </span>default<span class="w"> </span>via<span class="w"> </span><span class="m">192</span>.168.0.1
</code></pre></div>
<p>configuring openstack ml2 plugin with our vlan setup:</p>
<p>/etc/neutron/neutron.conf</p>
<div class="highlight"><pre><span></span><code><span class="nv">core_plugin</span><span class="w"> </span><span class="o">=</span>neutron.plugins.ml2.plugin.Ml2Plugin
</code></pre></div>
<p>Configuring the actual vlan's:</p>
<p>/etc/neutron/plugins/ml2/ml2_conf.ini</p>
<div class="highlight"><pre><span></span><code><span class="nv">type_drivers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>vxlan,gre,vlan
<span class="nv">network_vlan_ranges</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>vlan100:100:100
</code></pre></div>
<p>Creating the mapping between the vlan and the actual physical interface:</p>
<p>/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini</p>
<div class="highlight"><pre><span></span><code><span class="nv">bridge_mappings</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>vlan100:br-vlan
</code></pre></div>
<p>to get the metadata service usable on this flat network:</p>
<p>/etc/neutron/dhcp-agent.ini</p>
<div class="highlight"><pre><span></span><code><span class="nv">enable_isolated_metadata</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>True
</code></pre></div>
<p>restarting neutron-dhcp-agent</p>
<div class="highlight"><pre><span></span><code># openstack-service status neutron-dhcp-agent
</code></pre></div>
<p>Configuring the openstack networks on the all-in-one machine:</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="nx">source</span><span class="w"> </span><span class="nx">keystonerc_admin</span>
<span class="err">#</span><span class="w"> </span><span class="nx">neutron</span><span class="w"> </span><span class="nx">subnet</span><span class="o">-</span><span class="nx">list</span>
<span class="err">#</span><span class="w"> </span><span class="nx">neutron</span><span class="w"> </span><span class="nx">subnet</span><span class="o">-</span><span class="nx">delete</span><span class="w"> </span><span class="nx">ID</span>
<span class="err">#</span><span class="w"> </span><span class="nx">neutron</span><span class="w"> </span><span class="nx">net</span><span class="o">-</span><span class="nx">list</span>
<span class="err">#</span><span class="w"> </span><span class="nx">neutron</span><span class="w"> </span><span class="nx">net</span><span class="o">-</span><span class="nx">delete</span><span class="w"> </span><span class="nx">ID</span>
<span class="err">#</span><span class="w"> </span><span class="nx">neutron</span><span class="w"> </span><span class="nx">net</span><span class="o">-</span><span class="nx">create</span><span class="w"> </span><span class="nx">vlan100</span><span class="w"> </span><span class="o">--</span><span class="nx">shared</span><span class="w"> </span><span class="o">--</span><span class="nx">provider</span><span class="p">:</span><span class="nx">network_type</span><span class="w"> </span><span class="nx">vlan</span><span class="w"> </span><span class="o">--</span><span class="nx">provider</span><span class="p">:</span><span class="nx">segmentation_id</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="o">--</span><span class="nx">provider</span><span class="p">:</span><span class="nx">physical_network</span><span class="w"> </span><span class="nx">vlan100</span><span class="w"> </span><span class="o">--</span><span class="nx">router</span><span class="p">:</span><span class="kd">external</span>
<span class="err">#</span><span class="w"> </span><span class="nx">neutron</span><span class="w"> </span><span class="nx">subnet</span><span class="o">-</span><span class="nx">create</span><span class="w"> </span><span class="o">--</span><span class="nx">name</span><span class="w"> </span><span class="nx">vlan100</span><span class="w"> </span><span class="o">--</span><span class="nx">gateway</span><span class="w"> </span><span class="m m-Double">192.168.0.1</span><span class="w"> </span><span class="o">--</span><span class="nx">allocation</span><span class="o">-</span><span class="nx">pool</span><span class="w"> </span><span class="nx">start</span><span class="p">=</span><span class="m m-Double">192.168.0.150</span><span class="p">,</span><span class="nx">end</span><span class="p">=</span><span class="m m-Double">192.168.0.200</span><span class="w"> </span><span class="o">--</span><span class="nx">enable</span><span class="o">-</span><span class="nx">dhcp</span><span class="w"> </span><span class="o">--</span><span class="nx">dns</span><span class="o">-</span><span class="nx">nameserver</span><span class="w"> </span><span class="m m-Double">192.168.0.1</span><span class="w"> </span><span class="nx">vlan100</span><span class="w"> </span><span class="m m-Double">192.168.0.0</span><span class="o">/</span><span class="mi">24</span>
<span class="err">#</span><span class="w"> </span><span class="nx">neutron</span><span class="w"> </span><span class="nx">subnet</span><span class="o">-</span><span class="nx">update</span><span class="w"> </span><span class="o">--</span><span class="nx">host</span><span class="o">-</span><span class="nx">route</span><span class="w"> </span><span class="nx">destination</span><span class="p">=</span><span class="m m-Double">169.254.169.254</span><span class="o">/</span><span class="mi">32</span><span class="p">,</span><span class="nx">nexthop</span><span class="p">=</span><span class="m m-Double">192.168.0.151</span><span class="w"> </span><span class="nx">vlan100</span>
</code></pre></div>
<p>We do have a working setup right now if everything went well and you should be able to <a href="https://www.rdoproject.org/Running_an_instance">launch</a> an instance. To test ICMP traffic do not forget to enable a security group which allows this kind of traffic. Otherwise you couldn't use ping to test traffic.</p>
<p>Some useful commands:</p>
<div class="highlight"><pre><span></span><code>ovs-vsctl<span class="w"> </span>show<span class="w"> </span><span class="c1">#shows the openvswitch configuration</span>
ovs-ofctl<span class="w"> </span>dump-flows<span class="w"> </span>br-int<span class="w"> </span><span class="c1">#shows the flows to map an internal project tag to an actual vlan id</span>
brctl<span class="w"> </span>show<span class="w"> </span><span class="c1">#shows the linux bridge</span>
</code></pre></div>
<h3>persistent network configuration</h3>
<p>To keep your networking up and running after a reboot you should configure you bridges natively on the all-in-one instance:</p>
<p>/etc/sysconfig/network-scripts/ifcfg-eth0</p>
<div class="highlight"><pre><span></span><code><span class="nv">DEVICE</span><span class="o">=</span><span class="s2">"eth0"</span>
<span class="nv">ONBOOT</span><span class="o">=</span>yes
<span class="nv">OVS_BRIDGE</span><span class="o">=</span>br-vlan
<span class="nv">TYPE</span><span class="o">=</span>OVSPort
<span class="nv">DEVICETYPE</span><span class="o">=</span><span class="s2">"ovs"</span>
</code></pre></div>
<p>/etc/sysconfig/network-scripts/ifcfg-br-vlan</p>
<div class="highlight"><pre><span></span><code><span class="nv">DEVICE</span><span class="o">=</span>br-vlan
<span class="nv">BOOTPROTO</span><span class="o">=</span>none
<span class="nv">ONBOOT</span><span class="o">=</span>yes
<span class="nv">TYPE</span><span class="o">=</span>OVSBridge
<span class="nv">DEVICETYPE</span><span class="o">=</span><span class="s2">"ovs"</span>
</code></pre></div>
<p>/etc/sysconfig/network-scripts/ifcfg-br-vlan.100</p>
<div class="highlight"><pre><span></span><code><span class="nv">BOOTPROTO</span><span class="o">=</span><span class="s2">"none"</span>
<span class="nv">DEVICE</span><span class="o">=</span><span class="s2">"br-vlan.100"</span>
<span class="nv">ONBOOT</span><span class="o">=</span><span class="s2">"yes"</span>
<span class="nv">IPADDR</span><span class="o">=</span><span class="s2">"192.168.0.100"</span>
<span class="nv">PREFIX</span><span class="o">=</span><span class="s2">"24"</span>
<span class="nv">GATEWAY</span><span class="o">=</span><span class="s2">"192.168.0.1"</span>
<span class="nv">DNS1</span><span class="o">=</span><span class="s2">"192.168.0.1"</span>
<span class="nv">VLAN</span><span class="o">=</span>yes
<span class="nv">NOZEROCONF</span><span class="o">=</span>yes
<span class="nv">USERCTL</span><span class="o">=</span>no
</code></pre></div>
<p>Be sure to use the OVSBridge type and ovs DEVICETYPES otherwise it will not work..</p>
<p>Something we have on our todo is the <a href="http://docs.openstack.org/user-guide/cli_config_drive.html">configuration drive</a> setup. When using a configuration drive the metadata dhcp service could also be skipped and therefore possibly the whole openvswitch configuration could be passed by only using a provider network with a <a href="http://docs.openstack.org/networking-guide/deploy_scenario4b.html">linux bridge</a></p>
<p><a href="http://serverspec.org/">serverspec</a> were also written so the functionality of the puppet managed services are tested easily over and over to be sure the code is actually doing as it supposed to do.</p>Btrfs mount issue2015-09-07T19:00:00+02:002015-09-07T00:00:00+02:00Jantag:visibilityspots.org,2015-09-07:/btrfs-mount-issue.html<p>I decided to bootstrap my new machine with btrfs as filesystem instead of ext4 LVM volumes. By following the excellent arch-wiki <a href="https://wiki.archlinux.org/index.php/Btrfs">btrfs page</a> I successfully crafted a base system with sub volumes, limited on size and snapshots enabled.</p>
<p>Everything went fine, installed all the other stuff I needed, pulled in my data and was ready to go.</p>
<p>Obviously on that very moment disaster happened.. Due to an unexpected interrupt the journal went corrupt. When trying to boot I got stuck right after decrypted the disk failing to mount my btrfs root volume.</p>
<p>Uncool, unpleasant, .. I almost got insane..</p>
<p>So back …</p><p>I decided to bootstrap my new machine with btrfs as filesystem instead of ext4 LVM volumes. By following the excellent arch-wiki <a href="https://wiki.archlinux.org/index.php/Btrfs">btrfs page</a> I successfully crafted a base system with sub volumes, limited on size and snapshots enabled.</p>
<p>Everything went fine, installed all the other stuff I needed, pulled in my data and was ready to go.</p>
<p>Obviously on that very moment disaster happened.. Due to an unexpected interrupt the journal went corrupt. When trying to boot I got stuck right after decrypted the disk failing to mount my btrfs root volume.</p>
<p>Uncool, unpleasant, .. I almost got insane..</p>
<p>So back to the live usb arch Linux, decrypted my disk:</p>
<div class="highlight"><pre><span></span><code># cryptsetup luksOpen /dev/sda2
</code></pre></div>
<p>and tried to manually mount the root volume</p>
<div class="highlight"><pre><span></span><code># mount /dev/mapper/root /mnt
</code></pre></div>
<p>which resulted in some error like:</p>
<div class="highlight"><pre><span></span><code>parent transid verify failed on 109973766144 wanted 1823 found 1821
parent transid verify failed on 13891821568 wanted 540620 found 541176
parent transid verify failed on 13891821568 wanted 540620 found 541176
parent transid verify failed on 13891821568 wanted 540620 found 541176
parent transid verify failed on 13891821568 wanted 540620 found 541176
btrfs: open_ctree failed
</code></pre></div>
<p>Crawling through so many forum posts, stack overflow, blogposts a lot of solutions were suggested but none of them resulted in successfully mounting my brand new system..</p>
<p>After some time I could mount the filesystem in <a href="http://ram.kossboss.com/btrfs-restore-curropt-system/">recovery mode</a></p>
<div class="highlight"><pre><span></span><code># mkdir -p /mnt/root
# mount -o ro,recover /dev/mapper/root /mnt/root
</code></pre></div>
<p>So I decided to copy over all this data to an USB disk using rsync:</p>
<div class="highlight"><pre><span></span><code># mkdir -p /mnt/disk
# mount /dev/sdb1 /mnt/disk
# rsync -ah --progress /mnt/root /mnt/disk
</code></pre></div>
<p>Off course this took some time.. Once the data was copied over I followed the steps described by <a href="http://ram.kossboss.com/btrfs-transid-issue-explained-fix/">kossboss</a>:</p>
<div class="highlight"><pre><span></span><code># btrfs-zero-log /dev/mapper/root
# btrfsck --init-csum-tree /dev/mapper/root
# btrfsck --fix-crc /dev/mapper/root
# btrfsck --repair /dev/mapper/root
</code></pre></div>
<p>But no luck at all unfortunately</p>
<p>So I went online to the <a href="http://irc.lc/freenode/btrfs/">irc #btrfs</a> channel and explained my issue. A user named darkling reached out a hand and really made my day by suggesting to mount the filesystem as <a href="https://btrfs.wiki.kernel.org/index.php/Mount_options#Recovery">recovery</a> without the ro option.</p>
<div class="highlight"><pre><span></span><code># mount -orecovery /dev/mapper/root /mnt/root
</code></pre></div>
<p>And hell yeah I finally got it mounted, this recovery option did wrote a new working tree when I unmounted the filesystem so I could mount it in a normal way again!</p>
<div class="highlight"><pre><span></span><code># umount /mnt/root
# mount /dev/mapper/root /mnt/root
</code></pre></div>
<p>So I rebooted my system from live cd to hard drive and got my system back up and running!</p>
<p>Darkling really was my hero of the day!</p>Simple monitoring2015-05-26T19:00:00+02:002015-05-26T00:00:00+02:00Jantag:visibilityspots.org,2015-05-26:/simple-monitoring.html<p>As I already wrote about in the past I have a <a href="https://visibilityspots.com/raspberry-pi.html">raspberry pi</a> running at home. I do also have a VPS running somewhere on the interweb for an owncloud instance.</p>
<p>Being a sysadmin I wanted to know when my home devices become unreachable and when the owncloud instance is down. By mail in the first place if possible by sms message for free in the ideal world.</p>
<p>And guess what, I managed to reach the ideal world to monitor my instances. Over here I'll describe how I managed to do so.</p>
<h1>msmtp</h1>
<p>First of all you need to install …</p><p>As I already wrote about in the past I have a <a href="https://visibilityspots.com/raspberry-pi.html">raspberry pi</a> running at home. I do also have a VPS running somewhere on the interweb for an owncloud instance.</p>
<p>Being a sysadmin I wanted to know when my home devices become unreachable and when the owncloud instance is down. By mail in the first place if possible by sms message for free in the ideal world.</p>
<p>And guess what, I managed to reach the ideal world to monitor my instances. Over here I'll describe how I managed to do so.</p>
<h1>msmtp</h1>
<p>First of all you need to install and configure msmtp</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>pacman<span class="w"> </span>-Syu<span class="w"> </span>msmtp
</code></pre></div>
<p>The configuration is done in the /etc/msmtprc file, we use <a href="http://telenet.be">telenet</a> as ISP at home so therefore I used their smtp server details:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>vim<span class="w"> </span>/etc/msmtprc
<span class="w"> </span>defaults
<span class="w"> </span><span class="c1"># A providers service</span>
<span class="w"> </span>account<span class="w"> </span>telenet
<span class="w"> </span>from<span class="w"> </span>yourmailaddress
<span class="w"> </span>host<span class="w"> </span>out.telenet.be
<span class="w"> </span><span class="c1"># Set a default account</span>
<span class="w"> </span>account<span class="w"> </span>default<span class="w"> </span>:<span class="w"> </span>telenet
</code></pre></div>
<h1>mail-alert</h1>
<p>Next I wrote a bash script which I could pass some parameters so I could use it in many different situations to sent out mails.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>vim<span class="w"> </span>/usr/local/bin/mail-alert
<span class="w"> </span><span class="c1">#!/bin/bash</span>
<span class="w"> </span><span class="c1">#</span>
<span class="w"> </span><span class="c1"># Script which sends out mail using the given params</span>
<span class="w"> </span><span class="nv">RECIPIENT</span><span class="o">=</span><span class="nv">$1</span>
<span class="w"> </span><span class="nv">SUBJECT</span><span class="o">=</span><span class="nv">$2</span>
<span class="w"> </span><span class="nv">MESSAGE</span><span class="o">=</span><span class="nv">$3</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"To: </span><span class="nv">$RECIPIENT</span><span class="s2">"</span><span class="w"> </span>><span class="w"> </span>.mail
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"From: youremailaddress"</span><span class="w"> </span>>><span class="w"> </span>.mail
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Subject: </span><span class="nv">$SUBJECT</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>.mail
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span>>><span class="w"> </span>.mail
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$MESSAGE</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>.mail
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span>>><span class="w"> </span>.mail
<span class="w"> </span>cat<span class="w"> </span>.mail<span class="w"> </span><span class="p">|</span><span class="w"> </span>msmtp<span class="w"> </span><span class="nv">$RECIPIENT</span>
<span class="w"> </span>rm<span class="w"> </span>.mail<span class="w"> </span>-rf
</code></pre></div>
<p>This way you can easily sent out mails through the command line and therefore use this command in any of your scripts:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mail-alert<span class="w"> </span>recipient@domain.eu<span class="w"> </span><span class="s2">"Subject"</span><span class="w"> </span><span class="s2">"Your actual message"</span>
</code></pre></div>
<h1>monitor-lan</h1>
<p>So now I could actually send out mails through the command line I wrote a little bash script which performs some tests and based on the output triggers the mail-alert command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>vim<span class="w"> </span>/usr/local/bin/monitor-lan
<span class="w"> </span><span class="c1">#!/bin/bash</span>
<span class="w"> </span><span class="nv">HOSTS</span><span class="o">=</span><span class="s2">"8.8.4.4 gateway.ip.address"</span>
<span class="w"> </span><span class="nv">COUNT</span><span class="o">=</span><span class="m">4</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"==========================================================="</span><span class="w"> </span>>><span class="w"> </span>/tmp/monitor-lan.log
<span class="w"> </span><span class="k">for</span><span class="w"> </span>myHost<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="nv">$HOSTS</span>
<span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nv">count</span><span class="o">=</span><span class="k">$(</span>ping<span class="w"> </span>-c<span class="w"> </span><span class="nv">$COUNT</span><span class="w"> </span><span class="nv">$myHost</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'received'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span>-F<span class="s1">','</span><span class="w"> </span><span class="s1">'{ print $2 }'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{ print $1 }'</span><span class="k">)</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="nv">$count</span><span class="w"> </span>-eq<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$myHost</span><span class="s2"> was down at </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>/tmp/monitor-lan.log
<span class="w"> </span>mail-alert<span class="w"> </span>recipient@domain.eu<span class="w"> </span><span class="s2">"</span><span class="nv">$myHost</span><span class="s2"> is down"</span><span class="w"> </span><span class="s2">"This mail is to inform that host </span><span class="nv">$myHost</span><span class="s2"> is down (ping failed) at </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">"</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$myHost</span><span class="s2"> was alright ok at </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>/tmp/monitor-lan.log
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="k">done</span>
</code></pre></div>
<p>You can easily test this command by adding a non-reachable ip to the $HOSTS array and manually execute the monitor-lan command.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>monitor-lan
</code></pre></div>
<p>You should know have received a mail confirming the ip is down.</p>
<h1>sms</h1>
<p>As I already proclaimed I wanted this a step further and receive text messages on my cellphone instead of mails. I achieved this functionality using the service <a href="https://ifttt.com/wtf">ifttt</a>. In general it works as follows.</p>
<p>You create yourself an ifttt account and configure the <a href="https://ifttt.com/recipes/294447-sms-alerting-triggerd-by-mail">sms-alerting</a> recipe. Once that's done every mail you sent from the specified mail address using the #hashtag in the subject you configured in the mail channel to trigger@recipe.ifttt.com will trigger an sms to the mobile number you specified in the sms channel.</p>
<p>I used the #alert hashtag and therefore needed to reconfigure the monitor-lan script:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>vim<span class="w"> </span>/usr/local/bin/monitor-lan
<span class="w"> </span><span class="c1">#!/bin/bash</span>
<span class="w"> </span><span class="nv">PING_HOSTS</span><span class="o">=</span><span class="s2">"8.8.4.4 gateway.ip.address"</span>
<span class="w"> </span><span class="nv">COUNT</span><span class="o">=</span><span class="m">4</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"==========================================================="</span><span class="w"> </span>>><span class="w"> </span>/tmp/monitor-lan.log
<span class="w"> </span><span class="c1"># Reachable</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span>myHost<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="nv">$PING_HOSTS</span>
<span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nv">count</span><span class="o">=</span><span class="k">$(</span>ping<span class="w"> </span>-c<span class="w"> </span><span class="nv">$COUNT</span><span class="w"> </span><span class="nv">$myHost</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'received'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span>-F<span class="s1">','</span><span class="w"> </span><span class="s1">'{ print $2 }'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{ print $1 }'</span><span class="k">)</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="nv">$count</span><span class="w"> </span>-eq<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$myHost</span><span class="s2"> went down at </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>/tmp/monitor-lan.log
<span class="w"> </span>mail-alert<span class="w"> </span>trigger@recipe.ifttt.com<span class="w"> </span><span class="s2">"</span><span class="nv">$myHost</span><span class="s2"> is down #alert"</span><span class="w"> </span><span class="s2">"</span><span class="nv">$myHost</span><span class="s2"> went down (ping failed) at </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">"</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$myHost</span><span class="s2"> was alright at </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>/tmp/monitor-lan.log
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="k">done</span>
</code></pre></div>
<p>When you now add an unreachable ip and perform a manual test your should see after a while in the recipe log a trigger is been executed and the message should arrive on your mobile.</p>
<h1>remote checks</h1>
<p>Until now we only monitored the reachability of nodes through the icmp protocol. You could also perform more functional tests. Like for example login through ssh to a remote host and see if httpd is running.</p>
<p>Before adding the if statement to your existing monitor-lan script you should configure key based authentication between the pi and your remote host. This can easily been done by using the <a href="http://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/">ssh-copy-id</a> command.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[[</span><span class="w"> </span><span class="k">$(</span>su<span class="w"> </span>-<span class="w"> </span>username<span class="w"> </span>-c<span class="w"> </span><span class="s2">"ssh remote-node 'ps aux | grep httpd | grep -v grep | wc -l'"</span><span class="k">)</span><span class="w"> </span>!<span class="o">=</span><span class="w"> </span><span class="s2">"0"</span><span class="w"> </span><span class="o">]]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"httpd was up at </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>/tmp/monitor-lan.log
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"httpd went down at </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">"</span><span class="w"> </span>>><span class="w"> </span>/tmp/monitor-lan.log
<span class="w"> </span>mail-alert<span class="w"> </span>trigger@recipe.ifttt.com<span class="w"> </span><span class="s2">"Owncloud is down #alert"</span><span class="w"> </span><span class="s2">"Owncloud down (no httpd process running) at </span><span class="k">$(</span>date<span class="k">)</span><span class="s2">"</span>
<span class="w"> </span><span class="k">fi</span>
</code></pre></div>
<h1>cron</h1>
<p>Now everything is functional and in place you can configure a scheduled cronjob for it as root.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># crontab -e</span>
<span class="w"> </span><span class="c1"># Monitor LAN</span>
<span class="w"> </span>*/15<span class="w"> </span>*<span class="w"> </span>*<span class="w"> </span>*<span class="w"> </span>*<span class="w"> </span>/usr/local/bin/monitor-lan
</code></pre></div>
<p>You have now a rather easy peasy monitoring setup up and running which provides your the most basic monitoring for your different systems totally for free!</p>vagrant-setup2014-12-09T23:00:00+01:002014-12-09T00:00:00+01:00Jantag:visibilityspots.org,2014-12-09:/vagrant-setup.html<p>In this article I'll try to describe how I use vagrant in my daily tasks as an operations dude as well as I deployed it at one of our customers to help the developers focusing on the coding part rather than the operations part.</p>
<h1>Vagrant</h1>
<p>Since the beginning of my career at inuits I'm using <a href="https://vagrantup.com">vagrant</a> almost everyday. If I got payed every time I spin up a box I could have bought that tesla already some years ago! But unfortunately I'm not :)</p>
<p>For almost 99% of the use cases I use this nifty tool it's related to puppet. Writing …</p><p>In this article I'll try to describe how I use vagrant in my daily tasks as an operations dude as well as I deployed it at one of our customers to help the developers focusing on the coding part rather than the operations part.</p>
<h1>Vagrant</h1>
<p>Since the beginning of my career at inuits I'm using <a href="https://vagrantup.com">vagrant</a> almost everyday. If I got payed every time I spin up a box I could have bought that tesla already some years ago! But unfortunately I'm not :)</p>
<p>For almost 99% of the use cases I use this nifty tool it's related to puppet. Writing modules, testing out some configuration changes on a virtual machine first before pushing the changes to development, fighting with selinux, .. it are all crucial but changes that requires a lot of destroyments of boxes.</p>
<h1>Virtualization</h1>
<p>Vagrant uses some virtualization techniques to start stop and halt the different development machines. I listed the ones I use on a daily base or from time to time :)</p>
<h2>Virtualbox</h2>
<p>Like most of us I started using vagrant together with <a href="https://www.virtualbox.org/">virtualbox</a>. In the beginning I had a lot of issues when updating virtualbox, every time it was upgraded my vagrant environment failed to stay up and running.</p>
<p>Once I figured out most of my used vagrant boxes where reliant on the extension pack I never had any issue upgrading virtualbox anymore. The only thing I had to do was to upgrade the <a href="https://www.virtualbox.org/wiki/Downloads">extension pack</a> too.</p>
<p>It works great, but it's slow as hell when you spin up boxes again and again to test stuff out since it has to boot a whole vm every time. So about a year ago I started looking at some alternatives.</p>
<h2>LXC</h2>
<p>Looking around I found out about lxc containers. My interest was triggered at both <a href="linuxcon-edinburgh.html">linuxcon</a> and <a href="cloudcollab-amsterdam.html">CloudCollab</a> 2013 where some talks went about containers.</p>
<p>But I struggled configuring the lxc part on my previous fedora machine. It's one of the reasons I switched about a year ago. So I installed <a href="https://archlinux.org">ArchLinux</a> on my machine and started reading the related <a href="https://wiki.archlinux.org/index.php/Linux_Containers">wiki</a> pages.</p>
<p>Once I installed all necessary packages I went through the documentation of the <a href="https://github.com/fgrehm/vagrant-lxc">vagrant-lxc</a> plugin. There I got on a <a href="https://github.com/fgrehm/vagrant-lxc/wiki/Usage-on-Arch-Linux-hosts">page</a> describing the configuration steps for archlinux.</p>
<p>After following those configuration steps for the networking and DNS part of the lxc functionality I succeeded by creating my very first lxc centos based container:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>lxc-create<span class="w"> </span>-t<span class="w"> </span>centos<span class="w"> </span>-n<span class="w"> </span>centos
<span class="w"> </span>Host<span class="w"> </span>CPE<span class="w"> </span>ID<span class="w"> </span>from<span class="w"> </span>/etc/os-release:
<span class="w"> </span>This<span class="w"> </span>is<span class="w"> </span>not<span class="w"> </span>a<span class="w"> </span>CentOS<span class="w"> </span>or<span class="w"> </span>Redhat<span class="w"> </span>host<span class="w"> </span>and<span class="w"> </span>release<span class="w"> </span>is<span class="w"> </span>missing,<span class="w"> </span>defaulting<span class="w"> </span>to<span class="w"> </span><span class="m">6</span><span class="w"> </span>use<span class="w"> </span>-R<span class="p">|</span>--release<span class="w"> </span>to<span class="w"> </span>specify<span class="w"> </span>release
<span class="w"> </span>Checking<span class="w"> </span>cache<span class="w"> </span>download<span class="w"> </span><span class="k">in</span><span class="w"> </span>/var/cache/lxc/centos/x86_64/6/rootfs<span class="w"> </span>...
<span class="w"> </span>Cache<span class="w"> </span>found.<span class="w"> </span>Updating...
<span class="w"> </span>Loaded<span class="w"> </span>plugins:<span class="w"> </span>fastestmirror
<span class="w"> </span>Loading<span class="w"> </span>mirror<span class="w"> </span>speeds<span class="w"> </span>from<span class="w"> </span>cached<span class="w"> </span>hostfile
<span class="w"> </span>*<span class="w"> </span>base:<span class="w"> </span>be.mirror.eurid.eu
<span class="w"> </span>*<span class="w"> </span>extras:<span class="w"> </span>be.mirror.eurid.eu
<span class="w"> </span>*<span class="w"> </span>updates:<span class="w"> </span>be.mirror.eurid.eu
<span class="w"> </span>base<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">3</span>.7<span class="w"> </span>kB<span class="w"> </span><span class="m">00</span>:00
<span class="w"> </span>extra<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">3</span>.3<span class="w"> </span>kB<span class="w"> </span><span class="m">00</span>:00
<span class="w"> </span>update<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">3</span>.4<span class="w"> </span>kB<span class="w"> </span><span class="m">00</span>:00
<span class="w"> </span>Setting<span class="w"> </span>up<span class="w"> </span>Update<span class="w"> </span>Process
<span class="w"> </span>No<span class="w"> </span>Packages<span class="w"> </span>marked<span class="w"> </span><span class="k">for</span><span class="w"> </span>Update
<span class="w"> </span>Loaded<span class="w"> </span>plugins:<span class="w"> </span>fastestmirror
<span class="w"> </span>Cleaning<span class="w"> </span>repos:<span class="w"> </span>base<span class="w"> </span>extras<span class="w"> </span>updates
<span class="w"> </span><span class="m">0</span><span class="w"> </span>package<span class="w"> </span>files<span class="w"> </span>removed
<span class="w"> </span>Update<span class="w"> </span>finished
<span class="w"> </span>Copy<span class="w"> </span>/var/cache/lxc/centos/x86_64/6/rootfs<span class="w"> </span>to<span class="w"> </span>/var/lib/lxc/centos/rootfs<span class="w"> </span>...
<span class="w"> </span>Copying<span class="w"> </span>rootfs<span class="w"> </span>to<span class="w"> </span>/var/lib/lxc/centos/rootfs<span class="w"> </span>...
<span class="w"> </span>sed:<span class="w"> </span>can<span class="s1">'t read /etc/init/tty.conf: No such file or directory</span>
<span class="s1"> Storing root password in '</span>/var/lib/lxc/centos/tmp_root_pass<span class="s1">'</span>
<span class="s1"> Expiring password for user root.</span>
<span class="s1"> passwd: Success</span>
<span class="s1"> Container rootfs and config have been created.</span>
<span class="s1"> Edit the config file to check/enable networking setup.</span>
<span class="s1"> The temporary root password is stored in:</span>
<span class="s1"> '</span>/var/lib/lxc/centos/tmp_root_pass<span class="err">'</span>
<span class="w"> </span>The<span class="w"> </span>root<span class="w"> </span>password<span class="w"> </span>is<span class="w"> </span><span class="nb">set</span><span class="w"> </span>up<span class="w"> </span>as<span class="w"> </span>expired<span class="w"> </span>and<span class="w"> </span>will<span class="w"> </span>require<span class="w"> </span>it<span class="w"> </span>to<span class="w"> </span>be<span class="w"> </span>changed
<span class="w"> </span>at<span class="w"> </span>first<span class="w"> </span>login,<span class="w"> </span>which<span class="w"> </span>you<span class="w"> </span>should<span class="w"> </span><span class="k">do</span><span class="w"> </span>as<span class="w"> </span>soon<span class="w"> </span>as<span class="w"> </span>possible.<span class="w"> </span>If<span class="w"> </span>you<span class="w"> </span>lose<span class="w"> </span>the
<span class="w"> </span>root<span class="w"> </span>password<span class="w"> </span>or<span class="w"> </span>wish<span class="w"> </span>to<span class="w"> </span>change<span class="w"> </span>it<span class="w"> </span>without<span class="w"> </span>starting<span class="w"> </span>the<span class="w"> </span>container,<span class="w"> </span>you
<span class="w"> </span>can<span class="w"> </span>change<span class="w"> </span>it<span class="w"> </span>from<span class="w"> </span>the<span class="w"> </span>host<span class="w"> </span>by<span class="w"> </span>running<span class="w"> </span>the<span class="w"> </span>following<span class="w"> </span><span class="nb">command</span><span class="w"> </span><span class="o">(</span>which<span class="w"> </span>will
<span class="w"> </span>also<span class="w"> </span>reset<span class="w"> </span>the<span class="w"> </span>expired<span class="w"> </span>flag<span class="o">)</span>:
<span class="w"> </span>chroot<span class="w"> </span>/var/lib/lxc/centos/rootfs<span class="w"> </span>passwd
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>lxc-start<span class="w"> </span>-d<span class="w"> </span>-n<span class="w"> </span>centos
<span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>lxc-ls<span class="w"> </span>-f
<span class="w"> </span>NAME<span class="w"> </span>STATE<span class="w"> </span>IPV4<span class="w"> </span>IPV6<span class="w"> </span>AUTOSTART
<span class="w"> </span>---------------------------------------------------------------------------
<span class="w"> </span>centos<span class="w"> </span>RUNNING<span class="w"> </span><span class="m">10</span>.0.3.94<span class="w"> </span>-<span class="w"> </span>NO
<span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>lxc-console<span class="w"> </span>-n<span class="w"> </span>centos
<span class="w"> </span>CentOS<span class="w"> </span>release<span class="w"> </span><span class="m">6</span>.5<span class="w"> </span><span class="o">(</span>Final<span class="o">)</span>
<span class="w"> </span>Kernel<span class="w"> </span><span class="m">3</span>.17.4-1-ARCH<span class="w"> </span>on<span class="w"> </span>an<span class="w"> </span>x86_64
<span class="w"> </span>centos<span class="w"> </span>login:
</code></pre></div>
<p>It was a great feeling finally having that box up and running with internet available so I could update the box and install software on it.</p>
<h1>Vagrant-lxc</h1>
<p>Next up was to get such an lxc container up and running using the vagrant framework for my local development purposes.</p>
<h2>Manual crafted box</h2>
<p>So I first needed to get a centos minimal container with some vagrant tweaks on it. Looking around for documentation about this part I found out a helpful <a href="http://fabiorehm.com/blog/2013/07/18/crafting-your-own-vagrant-lxc-base-box/">blog post</a>.</p>
<p>But every time I tried to get that freshly build custom box up and running it failed. It was frustrating as hell! But when I figured out the logic behind the <a href="https://github.com/fgrehm/vagrant-lxc-base-boxes">vagrant-lxc-base-boxes</a> project I finally got to a working setup and could close the <a href="https://github.com/fgrehm/vagrant-lxc/issues/328">issue</a> myself.</p>
<p>So now I could configure a centos minimal lxc container for vagrant usage. But there are a lot of manual steps to perform if I want to keep that box up to date.</p>
<h2>Script crafted box</h2>
<p>When founding out about the <a href="https://github.com/fgrehm/vagrant-lxc-base-boxes">vagrant-lxc-base-boxes</a> project I tried to get up and running a centos box by following the documentation.</p>
<p>But once again it <a href="https://github.com/fgrehm/vagrant-lxc-base-boxes/issues/23">failed</a> big time. After some digging around I came to the conclusion the environment $PATH's of both archlinux the host and centos the guest are not the same and where the cause of this issue. Together with the missing <a href="https://github.com/visibilityspots/vagrant-lxc-base-boxes/commit/4274f0cdf593c26d92e438dd2b36a42f367691d7">ssh server</a> package I got the script working.</p>
<p>So I got a step further, I now could create a centos lxc vagrant box pretty easy myself.</p>
<h2>Script crafted box with puppet preinstalled</h2>
<p>Since I develop on puppet a lot, that minimal box needed at least the puppet software itself. The code of vagrant-lxc-base-boxes could do that but only for Debian based guests. So I <a href="https://github.com/visibilityspots/vagrant-lxc-base-boxes/commit/6fddcfec720bc9be80fe5620eabff88c27ae4637">refactored</a> that code for the centos boxes too.</p>
<p>And it worked out great! So great I'm sharing the box through <a href="https://vagrantcloud.com/visibilityspots/boxes/centos-6.x-puppet-3.x">vagrantcloud</a> so everyone can benefit of the usage of vagrant-lxc.</p>
<h1>Cloudstack</h1>
<p>At the Cloudstack Collaboration Conference 2014 in Budapest I followed this <a href="https://github.com/runseb/runseb.github.io/blob/master/ONEPAGE.md#vagrant">tutorial</a> of <a href="http://sebgoa.blogspot.be/">Sebastien Goasguen</a> where I successfully got a vagrant project up and running as a vm through cloudstack.</p>
<p>It felt great to get that vm up and running only through vagrant. It is like heaven, you can actually start coding in a vagrant-lxc box first, and test your code out on a so called real life production server only by using vagrant up!</p>
<h1>Production setup</h1>
<p>Since I got some great experiences with vagrant myself for developing puppet-modules the developers at the customer found out vagrant is really helpful tool in the actual developing part of a project. So we started with a tutorial of vagrant together with virtualbox.</p>
<p>In the initial stage of the vagrant implementation we all used some internet provisioned vagrant box. In the never ending process of improving the infrastructure we came to setup where a custom base box is provisioned by the operations team.</p>
<p>This base box is deployed using the same puppet code and therefore the same configuration as a production like server.</p>
<h2>Base box</h2>
<p>Every now and then that base box is updated by one of the operations people and placed on an accessible place for the developers.</p>
<p>The developers have configured vagrant together with the <a href="https://github.com/spil-ruslan/vagrant-box-updater">vagrant-box-updater</a> plugin. That way every time they bring up a vagrant project based on this base box a check will be performed if they are using the latest provisioned base box.</p>
<p>I do know it looks very like the golden images era. But I do believe also both team should have their focus on the right topic. Writing code for developers and managing infrastructure for operations.</p>
<p>Together with a well-thought deployment process (blog post coming soon) this works out really great.</p>
<h2>Virtualbox custom base box</h2>
<p>The base box is crafted based on <a href="https://vagrantcloud.com/search?utf8=%E2%9C%93&sort=&provider=&q=vstone">vStone</a> boxes and provisioned through puppet.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>vagrant<span class="w"> </span>init<span class="w"> </span>vStone/centos-6.x-puppet.3.x
<span class="w"> </span>$<span class="w"> </span>vagrant<span class="w"> </span>up
</code></pre></div>
<p>Once the box is up and running some manual tasks needs to be done</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>vagrant<span class="w"> </span>ssh
<span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>-s
<span class="w"> </span><span class="c1"># yum upgrade</span>
<span class="w"> </span><span class="c1"># yum groupinstall "Development Tools" -y</span>
<span class="w"> </span><span class="c1"># setenforce 1</span>
<span class="w"> </span><span class="c1"># vim /etc/sysconfig/selinux</span>
</code></pre></div>
<p>Some <a href="http://wiki.centos.org/HowTos/Virtualization/VirtualBox/CentOSguest">tweaks</a> for the virtualbox guest additions on centos 6.5</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="c1"># cd /usr/src/kernels/<kernel_release>/include/drm/</span>
<span class="w"> </span><span class="c1"># ln -s /usr/include/drm/drm.h drm.h</span>
<span class="w"> </span><span class="c1"># ln -s /usr/include/drm/drm_sarea.h drm_sarea.h</span>
<span class="w"> </span><span class="c1"># ln -s /usr/include/drm/drm_mode.h drm_mode.h</span>
<span class="w"> </span><span class="c1"># ln -s /usr/include/drm/drm_fourcc.h drm_fourcc.h</span>
<span class="w"> </span><span class="c1"># exit</span>
<span class="w"> </span>$<span class="w"> </span><span class="nb">exit</span>
</code></pre></div>
<p>Check virtualbox guest additions using the <a href="https://github.com/dotless-de/vagrant-vbguest">vagrant-vbguest</a>, that way the virtualbox guest additions are updated automatically every time you bring up the box</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>vagrant<span class="w"> </span>plugin<span class="w"> </span>install<span class="w"> </span>vagrant-vbguest
<span class="w"> </span>$<span class="w"> </span>vagrant<span class="w"> </span><span class="nb">suspend</span>
<span class="w"> </span>$<span class="w"> </span>vagrant<span class="w"> </span>up
</code></pre></div>
<p>Package the existing box with some default vagrant configuration</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>vim<span class="w"> </span><span class="s1">'''Vagrantfile.pkg'''</span>
<span class="w"> </span>Vagrant.configure<span class="o">(</span>VAGRANTFILE_API_VERSION<span class="o">)</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="p">|</span>config<span class="p">|</span>
<span class="w"> </span>config.vm.box_url<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"http://path/to/custom.box"</span>
<span class="w"> </span>config.vm.hostname<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"CUSTOMHOSTNAME"</span>
<span class="w"> </span>config.box_updater.autoupdate<span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">true</span>
<span class="w"> </span>end
</code></pre></div>
<p>Creation of the box</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>vagrant<span class="w"> </span>package<span class="w"> </span>--output<span class="w"> </span>custom.box<span class="w"> </span>--vagrantfile<span class="w"> </span>Vagrantfile.pkg
</code></pre></div>
<p>The custom.box is the actual box you need to provision on an accessible place for the development team.</p>
<h2>LXC custom base box</h2>
<p>For the lxc part a custom base box can also be created. To get all the processes done automatically I extended the vagrant-lxc-base-boxes project with an <a href="https://github.com/visibilityspots/vagrant-lxc-base-boxes/commit/92f14dd76e0e3f777b2a95d8643a45bbb0fff75c">own_box</a> feature.</p>
<p>That way you can easily create a vagrant box from an actively running lxc container you configured for your own needs.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>clone<span class="w"> </span>git@github.com:visibilityspots/vagrant-lxc-base-boxes.git
<span class="w"> </span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>vagrant-lxc-base-boxes
<span class="w"> </span>$<span class="w"> </span><span class="nv">ACTIVE_CONTAINER</span><span class="o">=</span>lxc-container-name<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>make<span class="w"> </span>own_box
</code></pre></div>
<p>To get the name of the running lxc-container you can use the lxc-ls command.</p>
<h1>Vagrantfile</h1>
<p>When starting to use both virtualization platforms next to each other I had created 2 identical Vagrantfile with the only difference the box.</p>
<p>But when reading through the docs I found out about you could also <a href="https://docs.vagrantup.com/v2/providers/configuration.html">override</a> some settings.</p>
<p>So I combined those 2 vagrantfiles into one. Depending of which platform you choose (virtualbox by default) the right box and settings are configured.</p>
<p>That way you only have to manage one Vagrantfile!</p>
<div class="highlight"><pre><span></span><code><span class="c1"># -*- mode: ruby -*-</span>
<span class="c1"># vi: set ft=ruby :</span>
<span class="no">VAGRANTFILE_API_VERSION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"2"</span>
<span class="no">Vagrant</span><span class="o">.</span><span class="n">configure</span><span class="p">(</span><span class="no">VAGRANTFILE_API_VERSION</span><span class="p">)</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="w"> </span><span class="n">config</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">define</span><span class="w"> </span><span class="s2">"default"</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="o">|</span><span class="n">default</span><span class="o">|</span>
<span class="w"> </span><span class="n">default</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">hostname</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"vagrant0"</span>
<span class="w"> </span><span class="n">default</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">provider</span><span class="w"> </span><span class="ss">:virtualbox</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="o">|</span><span class="n">virtualbox</span><span class="p">,</span><span class="w"> </span><span class="n">override</span><span class="o">|</span>
<span class="w"> </span><span class="n">override</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">box</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"vStone/centos-6.x-puppet.3.x"</span>
<span class="w"> </span><span class="n">override</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">network</span><span class="w"> </span><span class="ss">:forwarded_port</span><span class="p">,</span><span class="w"> </span><span class="ss">guest</span><span class="p">:</span><span class="w"> </span><span class="mi">80</span><span class="p">,</span><span class="w"> </span><span class="ss">host</span><span class="p">:</span><span class="w"> </span><span class="mi">8080</span>
<span class="w"> </span><span class="k">end</span>
<span class="w"> </span><span class="n">default</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">provider</span><span class="w"> </span><span class="ss">:lxc</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="o">|</span><span class="n">lxc</span><span class="p">,</span><span class="w"> </span><span class="n">override</span><span class="o">|</span>
<span class="w"> </span><span class="n">override</span><span class="o">.</span><span class="n">vm</span><span class="o">.</span><span class="n">box</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"visibilityspots/centos-6.x-puppet-3.x"</span>
<span class="w"> </span><span class="n">lxc</span><span class="o">.</span><span class="n">container_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'vagrant-container'</span>
<span class="w"> </span><span class="k">end</span>
<span class="w"> </span><span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<h1>Example project</h1>
<p>I abuse the <a href="https://github.com/visibilityspots/vagrant-yum-repo-server">vagrant-yum-repo-server</a> project to showcase the usage of vagrant in my world.</p>
<p>By using puppet, hiera, <a href="https://github.com/rasschaert/vagrant-puppet-stub">directory environments</a> thanks to a colleague <a href="http://www.kennyrasschaert.be">kenny</a> all with the vagrant puppet provisioner you actually get up and running a yum-repo-server yourself without big efforts.</p>
<p>As you can imagine that open up gates for developers AND operations cause they are all using the same puppet-tree so you should have control over the configuration of the box in all stages of the project.</p>
<h1>Vagrant-plugins I use</h1>
<ul>
<li><a href="https://github.com/klarna/vagrant-cloudstack">vagrant-cloudstack</a></li>
<li><a href="https://github.com/fgrehm/vagrant-global-status">vagrant-global-status</a></li>
<li><a href="https://github.com/smdahlen/vagrant-hostmanager">vagrant-hostmanager</a></li>
<li><a href="https://github.com/fgrehm/vagrant-lxc">vagrant-lxc</a></li>
<li><a href="https://github.com/dotless-de/vagrant-vbguest">vagrant-vbguest</a></li>
</ul>
<h1>Improvements</h1>
<p>Some improvements where I need more time for are</p>
<ul>
<li>automating the actual box creation by for example <a href="https://jenkins-ci.org">jenkins</a></li>
<li>auto update the base boxes processed like a development box using the <a href="ansible-orchestration.html">ansible</a> orchestration flow</li>
<li>..</li>
</ul>
<p>And of course the suggestions made by who know you? Cause this setup can always be improved, it will never been done..</p>Github mirroring2014-10-27T22:00:00+01:002014-10-27T00:00:00+01:00Jantag:visibilityspots.org,2014-10-27:/github-mirroring.html<p>As an enthusiastic open-source addict I use <a href="http://github.com">github</a> on a regularly base to share my knowledge with the world, to explore new software tools, to enhance software with new features, to fix bugs, to collaborate with others, and above all to live the open source way!</p>
<p>But I also have to admit that their are some disadvantages too, from time to time the availability, well lacks availability.., you have to pay for private repositories used for testing purposes and github enterprise can't be used publically anymore..</p>
<h1>Self-hosted git</h1>
<p>Using your own git instance makes your software less accessible, since like …</p><p>As an enthusiastic open-source addict I use <a href="http://github.com">github</a> on a regularly base to share my knowledge with the world, to explore new software tools, to enhance software with new features, to fix bugs, to collaborate with others, and above all to live the open source way!</p>
<p>But I also have to admit that their are some disadvantages too, from time to time the availability, well lacks availability.., you have to pay for private repositories used for testing purposes and github enterprise can't be used publically anymore..</p>
<h1>Self-hosted git</h1>
<p>Using your own git instance makes your software less accessible, since like the majority of the open-source software is available through github. But on the other hand, you can use as many private repositories as you want, you can protect your git server from the interweb by running it on your private network, you have control over the availability yourself, ...</p>
<p>You can set up your own <a href="http://git-scm.com/book/en/v1/Git-on-the-Server">git</a> server quite easy, together with some <a href="https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#GitJungle">frontend</a> it's all under <a href="http://www.visibilityspots.com/git-server.html">your own</a> control.</p>
<p>Their are some all in one systems available too, like <a href="http://redmine.org">redmine</a>, <a href="http://gitlab.com">gitlab</a>, <a href="http://chilliproject.org">chilliproject</a> and many others which combines the git repo functionality with some user management system in combination with or without project management and such.</p>
<h1>Mirroring</h1>
<p>So I looked for some setup where I could benefit from the github.com awareness and its features as well as the advantages of having your own git instance running.</p>
<p>After digging around I <a href="https://help.github.com/articles/duplicating-a-repository/">found</a> a mirroring solution which can be automated through for example a <a href="http://jenkinsci.org">jenkins</a> instance.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>clone<span class="w"> </span>--mirror<span class="w"> </span>git@private.gitinstance.org:localOrganisation/localRepo.git<span class="w"> </span>mirroring
<span class="w"> </span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>mirroring
<span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>remote<span class="w"> </span>set-url<span class="w"> </span>--push<span class="w"> </span>origin<span class="w"> </span>git@github.com:remoteOrg/remoteRepo.git
<span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>fetch<span class="w"> </span>-p<span class="w"> </span>origin
<span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>push<span class="w"> </span>--mirror
</code></pre></div>
<p>Setting up a mirror automated or not isn't really the biggest issue to solve. One of the things I was wondering about where the upstream pull requests.</p>
<h1>Merge upstream pull requests</h1>
<p>I wanted to have a way to pull those upstream pull requests into my private git repository. And get them synced afterwards with upstream. That way I could benefit of both github.com and a one.</p>
<p>And yes I found a way described by <a href="https://wincent.com/wiki/Setting_up_backup_(mirror)_repositories_on_GitHub#comment_10143">wincent</a>, to do so you have to go through following steps.</p>
<p>First of all go to the directory where your local git repository is located at.</p>
<p>Next up is to add a remote to your private git repo based on the user's upstream repo who made the pull request and fetch his repo into a local branch named to the user for example.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>localRepo
<span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>remote<span class="w"> </span>add<span class="w"> </span>USER<span class="w"> </span>git@github.com:USER/REPONAME.git
<span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>fetch<span class="w"> </span>USER
</code></pre></div>
<p>Once that's done you point your local repo to the master branch of the users branch you just fetched and test if the code is suiting your taste and everything is still up and running.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>checkout<span class="w"> </span>USER/master
</code></pre></div>
<p>When you are satisfied the code matches your standards and everything still works fine you can merge the pull request.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>checkout<span class="w"> </span>master
<span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>merge<span class="w"> </span>USER/master
<span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>push<span class="w"> </span>origin<span class="w"> </span>master
</code></pre></div>
<p>Once you merged the pull request successfully and you mirrored your local repo to your upstream one, you should see that the upstream pull request also has been automatically closed by your commit.</p>
<p>To keep your repos clean you should remove the freshly fetched branch from upstream out of your local repository</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>git<span class="w"> </span>remote<span class="w"> </span>rm<span class="w"> </span>USER
</code></pre></div>
<h1>Owner</h1>
<p>This whole article only works on git repositories owned by yourself or a git organization you are member of. What I'm still trying to solve is some automated way to keep some upstream github.com repository in sync with a local private git repository through a fork you made on github.com so you could create pull requests yourself to that software.</p>
<p>I am an open-source addict and I really want to make efforts to share features I wrote to existing upstream software by creating pull requests, but time has learned I do forget about creating them. So I should find myself a way how to keep my private local repository in sync with the forked one on github.com by mirroring. And some way I get notified about pieces of software ready to create a pull request.</p>
<p>Once those pull requests are merged by the upstream author they should be synchronized automatically with the fork and the private repo to keep them all in sync.</p>
<p>The reason why I want to automate this process is to not forget about sharing the code with the world cause I lack time, memory and resources to do so..</p>
<p>It's an excuse not worth an excuse and I really should make time for it! #blameingmyself</p>Ansible orchestration2014-10-21T23:00:00+02:002014-11-21T00:00:00+01:00Jantag:visibilityspots.org,2014-10-21:/ansible-orchestration.html<p>I do use <a href="https://docs.puppetlabs.com/#puppetpuppet">puppet</a> as our main configuration management tool. Together with <a href="https://docs.puppetlabs.com/#puppetdbpuppetdblatest">puppetdb</a> all our services are automatically configured from bottom to top.</p>
<p>And it rocks, getting automated as much as possible it is like easy as hell to get a server up and running. The only feature it lacked in my opinion is orchestration. I do know about <a href="http://puppetlabs.com/mcollective">collective</a> which is made for this purpose.</p>
<p>Only it's yet again using an agent which fails from time to time and eating resources which can be avoided. It's the same reason I don't use the puppet agent daemon but trigger puppet …</p><p>I do use <a href="https://docs.puppetlabs.com/#puppetpuppet">puppet</a> as our main configuration management tool. Together with <a href="https://docs.puppetlabs.com/#puppetdbpuppetdblatest">puppetdb</a> all our services are automatically configured from bottom to top.</p>
<p>And it rocks, getting automated as much as possible it is like easy as hell to get a server up and running. The only feature it lacked in my opinion is orchestration. I do know about <a href="http://puppetlabs.com/mcollective">collective</a> which is made for this purpose.</p>
<p>Only it's yet again using an agent which fails from time to time and eating resources which can be avoided. It's the same reason I don't use the puppet agent daemon but trigger puppet every time.</p>
<h1>orchestration</h1>
<p>We have puppet running every 15 minutes through cron, main reason is to pick up and install the latest software which has been deployed. The other reason puppet runs after installation is to make sure the configuration files were not manually manipulated and making sure necessary services are still running.</p>
<p>Using puppet for making sure services are running and configuration files are not being changed an hourly puppet run would be enough. Thing is for those deployment flows it's merely like polling. And I strongly hate polling jobs, 99% of the time they don't have to do anything. So to me it's just useless, a waste of time, energy and resources.</p>
<p>It meant that developers had to wait in worst case scenario 15 minutes before their changes where deployed on the development environment. Their changes were already processed by jenkins, packages are been made, deployed on the repository only waiting for puppet to install the latest version of them. Nobody complained, but in my opinion it was waaay too long!</p>
<p>By running puppet immediately after the package is been deployed to the repository the right order of installing, configuring and restarting the necessary services can be executed. This will gain time for deployments next to some hourly puppet cron jobs which are running just to be sure no configuration has changed manually and the services are still running.</p>
<h1>ansible</h1>
<p>So I started looking at some solution where I could trigger a puppet run on the hosts configured the software through puppet in the right environment as soon as the package is deployed to the proper repository through jenkins.</p>
<p>At first I looked into the ssh jenkins plugin, it works but has one big disadvantage. You have to configure ssh credentials for every host in jenkins and therefore you can't use abstract jenkins flows cause you need to configure in each job the specific ssh credentials.</p>
<p>I looked further and came across <a href="http://www.ansible.com">ansible</a>. You don't have to configure a client on every host, neither you have to configure a per server based jenkins configuration to get it working. It was a blessing, the only things you have to do is creating a user, his public ssh key and grant him sudo rights on every server. This can easily be done through puppet!</p>
<h1>static inventory</h1>
<p>At first I crawled through our <a href="http://www.theforeman.org">foreman</a> instance and copied over the nodes into 2 groups, development and production, the puppet environments. I also configured some stuff like ssh port and user. I refused to configure the root pw in some plain text file on the jenkins node. That's not safe at all in my opinion, instead I created an ssh key pair and distributed the public key on all servers.</p>
<p>In my fight to automate as much as possible this wasn't the most efficient way of using the inventory. Every time you removed or added a node you had to reconfigure it yourself manually in the first place. Beside the manual intervention you also have to take note how you are gonna perform that manual action? Manipulating configuration data on the production machine is not done, using a git repository which you package or adding them to puppet, which both sounds wrong. The first because it's overkill the second because it's rather data over configuration.</p>
<h1>dynamic inventory</h1>
<p>In my quest I got pointed to a <a href="https://github.com/EchoTeam/ansible-plugins">python</a> script by a colleague. Unfortunately the script isn't straight forward and the 'maintainers' hides themselves behind their footer:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="nv">Notice</span>:<span class="w"> </span><span class="nv">The</span><span class="w"> </span><span class="nv">puppetdb</span><span class="w"> </span><span class="nv">inventory</span><span class="w"> </span><span class="nv">plugin</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">not</span><span class="w"> </span><span class="nv">quite</span><span class="w"> </span><span class="nv">generic</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">moment</span>.<span class="w"> </span><span class="nv">Use</span><span class="w"> </span><span class="nv">more</span><span class="w"> </span><span class="nv">as</span><span class="w"> </span><span class="nv">an</span><span class="w"> </span><span class="nv">example</span>.
</code></pre></div>
<p>Once I found out about the <a href="http://docs.ansible.com/developing_inventory.html">inventory</a> part of ansible I knew what I was looking for and saw the light by an <a href="https://blog.codecentric.de/en/2014/09/use-ansible-remote-executor-puppet-environment/">article</a> on cedecentric.de. Their was only one issue, my jenkins host which needs ansible to run isn't my puppetmaster and therefore can't list the signed certificates as used in his script.</p>
<p>But I am using <a href="https://docs.puppetlabs.com/puppetdb/latest/index.html">puppetdb</a>, and puppetdb has a great <a href="https://docs.puppetlabs.com/puppetdb/2.2/api/index.html">API</a>. So I could take advantage of it by using this great API, melting it down into an inventory script and using the json generated output through ansible.</p>
<p>So I started modifying the code example I found on codecentrec and got it working by writing a <a href="https://github.com/visibilityspots/ansible-puppet-inventory">puppetdb.sh</a> dynamic inventory script. Together with the <a href="https://github.com/visibilityspots/puppet-ansible">puppet-ansible</a> module it even got automated too!</p>
<h1>adding it to ansible-core</h1>
<p>I went to the <a href="http://events.linuxfoundation.org/events/cloudstack-collaboration-conference-europe">Cloudstack Collaboration Conference</a> in Budapest where I followed a <a href="https://github.com/runseb/runseb.github.io/blob/master/ONEPAGE.md">tutorial</a> by <a href="http://sebgoa.blogspot.hu/">Sebastien Goasguen</a>.</p>
<p>It turned out he wrote an ansible apache-libcloud inventory script and tried to pushing it in to the ansible core. This inspired my to rewrite my bash script in python so it could be added to ansible-core too.</p>
<p>After fooling around a bit in python I used the <a href="https://github.com/puppet-community/pypuppetdb">pypuppetdb</a> library so I don't have to make all the API calls natively myself through urllib request. And it turned out quite fine and I got it up and running in my setup. So those days I'm waiting on feedback from the ansible community to my <a href="https://github.com/ansible/ansible/pull/9593">pull request</a> so everyone can benefit of the joy between ansible and puppet.</p>
<h1>still need some attention</h1>
<p>I need some time to look which processes it takes to run a command through ansible so I could specify more clear the sudoers file.</p>
<p>Also the environments should be more abstract in my puppetdb.sh script without having to manually adapt the necessary puppetdb query files.</p>
<h1>drinking cocktails</h1>
<p>From now on it only takes less than 5 minutes to push your code, get it through jenkins tests into a package on an apt or yum repository got pulled into a repository and deploy it through puppet using ansible on the development servers. All without any manual action, without any cron job all automated, glued the pieces together.</p>
<p>I'll dig deeper into the whole deployment process later on, when I found time between drinking cocktails, looking at my daughter and living the dream.</p>Wifi QR code2014-08-28T22:00:00+02:002014-08-28T00:00:00+02:00Jantag:visibilityspots.org,2014-08-28:/wifi-qr-code.html<p>To make the process of connecting to our local wifi at home a bit less complex I decided to create a qr code for it. That way people can easily use their camera of their smartphone to connect to our network without typing in the WPA key.</p>
<p>So I looked on the net for a qr generator and started by typing in our SSID, when realizing it can't be secure to fill in our wpa key too.. It may be a bit paranoia but well I don't trust anything on the interweb most of the time.</p>
<p>Doing some further research …</p><p>To make the process of connecting to our local wifi at home a bit less complex I decided to create a qr code for it. That way people can easily use their camera of their smartphone to connect to our network without typing in the WPA key.</p>
<p>So I looked on the net for a qr generator and started by typing in our SSID, when realizing it can't be secure to fill in our wpa key too.. It may be a bit paranoia but well I don't trust anything on the interweb most of the time.</p>
<p>Doing some further research I found out about <a href="http://fukuchi.org/works/qrencode/">qrencode</a> a command line tool which can be used to generate many different QR codes.</p>
<p>For installation in archlinux you can use pacman:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>pkgfile<span class="w"> </span>qrencode
<span class="w"> </span>extra/qrencode
<span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>pacman<span class="w"> </span>-Syu<span class="w"> </span>qrencode
</code></pre></div>
<p>Once installed you can generate a qr code consisting of your wifi data base on some standardized <a href="https://github.com/zxing/zxing/wiki/Barcode-Contents">barcode contents</a>.</p>
<p>When figured out what that content had to be I generated an svg file of our QR code:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>qrencode<span class="w"> </span>-t<span class="w"> </span>SVG<span class="w"> </span>-o<span class="w"> </span>wifi-code.svg<span class="w"> </span><span class="s2">"WIFI:S:SSID-OF-YOUR-WLAN;T:WPA2;P:YOUR-WPA2-KEY;;"</span>
</code></pre></div>
<p>You now should be able to print that svg file with your SSID connection information you could use to connect to your network by simple scanning the code from your smartphone using a <a href="https://play.google.com/store/search?q=barcode">barcode scanner</a> of your choice.</p>
<p>And by not using the internet but your local machine there is no chance anyone is storing that data in some dark database ;)</p>Rsnapshot backup2014-07-16T19:00:00+02:002015-06-15T00:00:00+02:00Jantag:visibilityspots.org,2014-07-16:/rsnapshot-backup.html<p>Some weeks ago I removed some files on my system I wish I didn't cause I still needed them.</p>
<p>Since I use some <a href="https://visibilityspots.com/dropbox.html">encrypted containers in dropbox</a> I figured I could recover them from this nifty service. But in the logs of dropbox those files are obviously also encrypted. So that was a no go.</p>
<p>Therefore I started looking for backup solutions. The first one I tried was <a href="http://backintime.le-web.org/">backintime</a>, very easy to use, clean GUI interface but failed multiple times in restoring some files.</p>
<p>Looking for a more decent piece of software I found <a href="http://rsnapshot.org">rsnapshot</a></p>
<p>And hell I like it …</p><p>Some weeks ago I removed some files on my system I wish I didn't cause I still needed them.</p>
<p>Since I use some <a href="https://visibilityspots.com/dropbox.html">encrypted containers in dropbox</a> I figured I could recover them from this nifty service. But in the logs of dropbox those files are obviously also encrypted. So that was a no go.</p>
<p>Therefore I started looking for backup solutions. The first one I tried was <a href="http://backintime.le-web.org/">backintime</a>, very easy to use, clean GUI interface but failed multiple times in restoring some files.</p>
<p>Looking for a more decent piece of software I found <a href="http://rsnapshot.org">rsnapshot</a></p>
<p>And hell I like it. It's not a GUI interface which is a plus for me, it don't use a huge amount of disk space for all the backups, instead it uses hard links between non changed files and it's completely indepented and automizable by cron.</p>
<p>So I bought myself a <a href="https://www.lacie.com/products/product.htm?id=10609">lacie slim P9223 SSD</a> external USB 3.0 disk from about 120GB only used for backing up my system.</p>
<p><img alt="lacie" src="../../images/backup/lacie-p9223-slim.png"></p>
<h1>Encryption</h1>
<p>Since I want my data to be back upped in a secure way I chose to encrypt the entire disk. I will only use it for this backup purpose so it doesn't matter. To do so I followed a <a href="http://blog.abhijeetr.com/2012/06/encrypt-partition-luks-cryptsetup-on.html">tutorial</a> I found on the net which describes very clearly every step in the process to achieve an encrypted disk using <a href="https://code.google.com/p/cryptsetup/">cryptsetup</a>.</p>
<p>The data I backup are the decrypted encfs containers from within my dropbox instance. That way dropbox contains my data in an encrypted way so I can synchronize them between multiple machines and I still have a backup on my external hard disk in case of disaster happens.</p>
<h1>UDEV rule</h1>
<p>Every time an usb device is connected it is attached to another random /dev/sdX pointer. Since I wrote a little script to mount/umount this encrypted disk I needed a persistent pointer. Using some nifty <a href="https://bbs.archlinux.org/viewtopic.php?id=134705">udev rule</a> based on the serial of the device this issue is solved by always attaching this disk to the /dev/backup pointer..</p>
<p>Get your devices short serial:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>udevadm<span class="w"> </span>info<span class="w"> </span>-q<span class="w"> </span>all<span class="w"> </span>-n<span class="w"> </span>/dev/sdX1<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>SERIAL_SHORT
<span class="w"> </span>E:<span class="w"> </span><span class="nv">ID_SERIAL_SHORT</span><span class="o">=</span><span class="m">00000000000000000000</span>
</code></pre></div>
<p>Write the udev rule itself to attach the device to the /dev/backup pointer:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>vim<span class="w"> </span>/etc/udev/rules.d/66-persistent-backup-disk.rules
<span class="w"> </span><span class="nv">KERNEL</span><span class="o">==</span><span class="s2">"sd?1"</span>,<span class="w"> </span><span class="nv">SUBSYSTEMS</span><span class="o">==</span><span class="s2">"block"</span>,<span class="w"> </span>ENV<span class="o">{</span>ID_SERIAL_SHORT<span class="o">}==</span><span class="s2">"00000000000000000000"</span>,<span class="w"> </span><span class="nv">SYMLINK</span><span class="o">+=</span><span class="s2">"backup"</span>
</code></pre></div>
<p>That way the device is always attached to /dev/backup and could therefore be used to decrypt and mount:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>sudo<span class="w"> </span>cryptsetup<span class="w"> </span>open<span class="w"> </span>/dev/backup<span class="w"> </span>backup
<span class="w"> </span>sudo<span class="w"> </span>mount<span class="w"> </span>/dev/mapper/backup<span class="w"> </span>/mnt/backup/
</code></pre></div>
<p>or umounted and encrypted:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>sudo<span class="w"> </span>umount<span class="w"> </span>/mnt/backup
<span class="w"> </span>sudo<span class="w"> </span>cryptsetup<span class="w"> </span>luksClose<span class="w"> </span>backup
</code></pre></div>
<h1>Rsnapshot</h1>
<p>I read the wiki page of <a href="https://wiki.archlinux.org/index.php/Rsnapshot">rsnapshot</a> from archlinux and followed their configuration instructions and adapted those to my own preferences. After this configuration has been saved and tested I added some cron magic to automate those backups during my lunch break.</p>
<div class="highlight"><pre><span></span><code> # Backup schedule (rsnapshot)
0 <span class="gs">*/3 *</span> * <span class="gs">* /usr/bin/rsnapshot hourly</span>
<span class="gs"> 30 12 *</span> * <span class="gs">* /usr/bin/rsnapshot daily</span>
<span class="gs"> 45 12 *</span> * 1 /usr/bin/rsnapshot weekly
0 13 1 * 1 /usr/bin/rsnapshot monthly
</code></pre></div>
<h1>Fsck</h1>
<p>I ran into some ext4 troubles on my encrypted filesystem, it took me a while to figure out how I could resolve them. You had to open the encrypted container without actually mounting the device.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>sudo<span class="w"> </span>cryptsetup<span class="w"> </span>luksOpen<span class="w"> </span>/dev/sdX#<span class="w"> </span>SOMENAME
</code></pre></div>
<p>Once that's done you could use fsck to manually fix the filesystem.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>sudo<span class="w"> </span>fsck.ext4<span class="w"> </span>/dev/mapper/SOMENAME
</code></pre></div>
<p>or let fsck automatically fix the filesystem.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>sudo<span class="w"> </span>fsck.ext4<span class="w"> </span>-y<span class="w"> </span>/dev/mapper/SOMENAME
</code></pre></div>
<p>If everything got fixed you could mount the partition and use the fixed filesystem again.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>sudo<span class="w"> </span>mount<span class="w"> </span>/mnt/backup
</code></pre></div>
<p>So from now on my data is back upped on an encrypted USB hard disk without using any fancy pansy GUI interface :)</p>
<p>Hence feel free to comment if there are some quirks you tackled another way around!</p>Nest2014-05-24T23:30:00+02:002014-06-09T00:00:00+02:00Jantag:visibilityspots.org,2014-05-24:/nest.html<p>A few years ago I discovered the <a href="http://nest.com">nest</a> thermostat. It looked nice, is connected to the internet, self learning but a bit expensive and not so much documentation if it will work in Belgium nor with the boiler we have installed (Vaillant TurboTec).</p>
<p>Doing some research on the internet I figured that they have updated their system software and are supporting European countries including Belgium. A solution for the power has also been found by an ICY converter so it could easily be implemented in setups where no power is transferred over those wires.</p>
<p>So I had only to tackle …</p><p>A few years ago I discovered the <a href="http://nest.com">nest</a> thermostat. It looked nice, is connected to the internet, self learning but a bit expensive and not so much documentation if it will work in Belgium nor with the boiler we have installed (Vaillant TurboTec).</p>
<p>Doing some research on the internet I figured that they have updated their system software and are supporting European countries including Belgium. A solution for the power has also been found by an ICY converter so it could easily be implemented in setups where no power is transferred over those wires.</p>
<p>So I had only to tackle the price, you could order one using the official nest store of the <a href="http://store.nest.com/uk/">UK</a> but then you would pay about € 300 euro's for only the device. And that was still too pricy for me.</p>
<p>Luckily you can find them for about € 190 on <a href="http://www.amazon.com/gp/offer-listing/B009GDHYPQ/ref=sr_1_1_olp?ie=UTF8&qid=1401011588&sr=8-1&keywords=nest&condition=new">amazon</a>, you have to take into account the shipping and handling fee, € 8 and the import fees € 42. Together with the <a href="http://www.eco-logisch.be/Details.asp?ProductID=3484&category=115">ICY converter</a> which is about € 50 you have the whole setup for about € 290.</p>
<p>So you could save some money by ordering with amazon!</p>
<p>Convincing myself it's worth the money, and it could save me some on the very long term I took the plunge and ordered one so I could enjoy those geeky features. (Doing so my girlfriend is convinced I'm a nerd, especially because I'm also blogging about it right now)</p>
<p>It only took a couple of days to have those 2 packages delivered and I could start playing around with it.</p>
<h1>Unboxing</h1>
<p>The nest comes into an hipster sleek box.</p>
<p><img alt="nest-boxes" src="../../images/nest/nest-boxes.jpg"></p>
<p>It's almost the same feeling as unboxing the ipod I bought about 10 years ago. Opening the box the first thing which you discover is the nifty looking product itself.</p>
<p><img alt="nest-unboxing-one" src="../../images/nest/nest-unboxing-one.jpg"></p>
<p>Underneath the nest display a whole bunch of stuff is packed, a paper guide, cool looking screw driver, a couple of mount plates, some screws and the base plate to connect the wires too.</p>
<p><img alt="nest-unboxing-all" src="../../images/nest/nest-unboxing-all.jpg"></p>
<h1>Initialization</h1>
<p>First thing I did was charging the display using a micro USB cable, so I could start configuring it afterwards without being disrupted due to a low battery.</p>
<p><img alt="nest-charging" src="../../images/nest/nest-charging.jpg"></p>
<p>After a night of charging I started configuring the thermostat by first connecting it to our wireless network (using a WPA2 encryption) without any hassle it got connected and started downloading the latest updates</p>
<p><img alt="nest-updating" src="../../images/nest/nest-updating.png"></p>
<p>After the updated had been downloaded and got installed the device rebooted itself and was ready for the real work.</p>
<h1>Old thermostat</h1>
<p>Before I could get started I had to remove our current thermostat, a Honeywell CM907. I went ahead after I shut down power from our boiler which is connected to the thermostat.</p>
<p><img alt="honeywell" src="../../images/nest/honeywell.jpg"></p>
<p>This can be done easily once I figured out how, you have to remove the cover, battery lid and batteries. In the battery compartment I marked two little arrows which indicates being clips you can unlock using a screw driver. Once I got ejected that control panel I got to the wiring underneath.</p>
<p><img alt="disassembling-desk" src="../../images/nest/disassembling-desk.jpg"></p>
<p>I wrote down the current wired setup on a sticker and putted it on that plastic base mount. That way I could easily roll back if anything will go wrong in the near or far away future.</p>
<p><img alt="dismantled-honeywell" src="../../images/nest/dismantled-honeywell.jpg"></p>
<h1>Wiring</h1>
<h2>ICY connection</h2>
<p>Doing my research on the net I came to the issue of powering your nest device. In Europe most devices are powered by batteries, unfortunately the nest device doesn't. To solve this issue you could use a so called ICY converter. I had to cut the wires coming from my boiler to the thermostat itself.</p>
<p><img alt="cutting-wires" src="../../images/nest/cutting-wires.jpg"></p>
<p>Next step was to connect the wires from the boiler to the CV clips on the converter and the wires going to the thermostat on the Th clips.</p>
<p><img alt="ICY-converter" src="../../images/nest/ICY-convertor.jpg"></p>
<p>After connecting all the wires I did not yet plugged the converter in. I first installed the mount plate for the base of the nest before powering on all the electrical devices.</p>
<p><img alt="ICY-connector" src="../../images/nest/ICY-connector.jpg"></p>
<h2>Mountplate</h2>
<p>After disassembling my old thermostat I discovered that the wall has some holes and missing paint behind that old mount plate. Luckily you could use the enclosed base cover to hide those imperfections. Once I drilled the holes, screwed the base plate onto the wall I could connect the wires. As you can see on the picture you have to connect on on the W1 port and one on the Rh port.</p>
<p><img alt="nest-wires" src="../../images/nest/nest-wires.png"></p>
<p>When I then attached the display, powered on the boiler and plugged in the converter the nest asked configuration of the heating equipment I first got the <a href="http://support.nest.com/#troubleshooting/e24">E24</a> error which means no power is detected coming to the nest device. I switched the wires from W1 and Rh and repowered the boiler. Now I got the <a href="http://support.nest.com/#troubleshooting/n23">N23</a> looking at the support pages it's indicating you are using an unusual configuration but my boiler picked up signals coming from the nest thermostat.</p>
<h1>Configuration</h1>
<p>I went to all the configuration steps, connecting to the wifi, adding to my online nest account, setting the auto-away options, ... and many more. Once that's done you could see on the screen that I turned the nest to it's minimal, 9° C and that on the moment of taking the picture it was 26° C inside the nest device.</p>
<p><img alt="nest-working" src="../../images/nest/nest-working.png"></p>
<p>Since I took the decision to install this thermostat in the summer when we don't need the CV system I could not yet fully test the device. I didn't wanted to take the risk not having a working CV when it was cold.</p>
<p><img alt="nest-off" src="../../images/nest/nest-off.png"></p>
<p>So I had to turn it off for now and will come back with a review of the usage in most likely december when the winter has taken his start over here in Belgium.</p>
<h1>Statistics</h1>
<p>Doing some research about applications for the nest thermostat I discovered a tool called <a href="https://github.com/chriseng/nestgraph">nest-graph</a>. This tool collects data from the nest device using the unofficial <a href="https://github.com/gboudreau/nest-api">nest-api</a> through php.</p>
<p>I forked the nest-graph repository to make some <a href="https://github.com/visibilityspots/nestgraph.git">customizations</a> like using Celsius instead of Fahrenheit, adding the nest-api-master as a git submodule, changing the database setup and some ignoration of configuration files.</p>
<p>Cause I have the availability of a mysql database on my <a href="http://one.com">one.com</a> account I decided to collect the data into that database and running the nestgraph service onto that website. Some modifications had to be made to the dbsetup file and adding the login parameters to inc/config.php file.</p>
<p>After successfully running the command</p>
<div class="highlight"><pre><span></span><code><span class="x"> $ php test.php</span>
</code></pre></div>
<p>Once that test is succeeded you could use</p>
<div class="highlight"><pre><span></span><code><span class="x"> $ php insert.php</span>
</code></pre></div>
<p>to fetch the real time data from your nest device and import them into your mysql database.</p>
<p>From my <a href="../raspberry-pi.html">raspberry-pi</a> I configured a cron job to log in into the my one.com hosting account using ssh and start the import job every 5 minutes.</p>
<div class="highlight"><pre><span></span><code> <span class="gs">*/5 *</span> * <span class="gs">* *</span> ssh ONE.COM-URL "/bin/rm -f /tmp/nest_php_* ; /usr/bin/php /path/to/nestgraph/insert.php > /dev/null"
</code></pre></div>
<p>Last but not least I added some security to this service using <a href="http://one-docs.com/tools/htaccess/">htaccess</a> so all this information isn't publicly available. Now I only have to wait to winter so I can finally start monitoring our heating usage!</p>
<p><img alt="nest-graph" src="../../images/nest/nest-graph.png"></p>
<h1>Resources</h1>
<ul>
<li><a href="http://www.fousa.be/blog/nest-thermostat">fousa.be</a></li>
<li><a href="http://promo-code.be/nest-thermostaat/">promo-code.be</a> (dutch)</li>
<li><a href="http://home.howstuffworks.com/nest-learning-thermostat.htm">how-stuff-works</a></li>
<li><a href="https://github.com/gboudreau/nest-api">nest-api</a></li>
<li><a href="https://github.com/chriseng/nestgraph">nest-graph</a></li>
<li><a href="http://blog.spark.io/2014/01/17/open-source-thermostat/">open-source</a> nest alternative</li>
</ul>Raspberry pi setup2014-05-16T19:00:00+02:002014-08-04T00:00:00+02:00Jantag:visibilityspots.org,2014-05-16:/raspberry-pi.html<p>Since I discovered the joy of linux servers over desktop distributions a few years ago I revived an old portable and promoted him to be my home server.</p>
<p>Connected him our router in the little storage room on a top shelf gathering dust I could test, configure, break (and pass sleep) a huge variety of open-source software.</p>
<p>Many of those adventures I also used to provide my blog with content. After a while I figured this setup isn't really needed to be powered on 24h a day 7 days a week. So I bought myself a <a href="http://www.raspberrypi.org/">raspberry pi</a> which would …</p><p>Since I discovered the joy of linux servers over desktop distributions a few years ago I revived an old portable and promoted him to be my home server.</p>
<p>Connected him our router in the little storage room on a top shelf gathering dust I could test, configure, break (and pass sleep) a huge variety of open-source software.</p>
<p>Many of those adventures I also used to provide my blog with content. After a while I figured this setup isn't really needed to be powered on 24h a day 7 days a week. So I bought myself a <a href="http://www.raspberrypi.org/">raspberry pi</a> which would cover the basic functionalities I needed to be online as much as possible without the need of a subscription for a VPS or dedicated server in one fancy pansy data center.</p>
<p>For the operating system I didn't choose for the default <a href="http://www.raspbian.org">raspbian</a>, mainly because I don't need a graphical interface. So I headed over to archlinux arm, also called <a href="http://archlinuxarm.org/platforms/armv6/raspberry-pi">alarm</a>.</p>
<p>The <a href="http://archlinuxarm.org/platforms/armv6/raspberry-pi#qt-platform_tabs-ui-tabs2">installation</a> is quite straight forward. After that I installed and configured some other stuff on it to gather my ultimate personal little mini server:</p>
<ul>
<li><a href="https://wiki.archlinux.org/index.php/vim">vim</a></li>
<li><a href="https://wiki.archlinux.org/index.php/Secure_Shell">ssh</a></li>
<li><a href="https://wiki.archlinux.org/index.php/Secure_Shell#SSH_alternative:_Mobile_Shell_-_responsive.2C_survives_disconnects">mosh</a></li>
<li><a href="https://wiki.archlinux.org/index.php/Noip">noip</a></li>
<li><a href="https://wiki.archlinux.org/index.php/GNU_Screen">screen</a></li>
<li><a href="https://wiki.archlinux.org/index.php/yaourt">yaourt</a></li>
<li><a href="https://wiki.archlinux.org/index.php/pkgfile">pkgfile</a></li>
<li><a href="https://www.archlinux.org/packages/extra/any/bash-completion/">bash-completion</a></li>
<li><a href="https://www.archlinux.org/packages/community/x86_64/ncdu/">ncdu</a></li>
<li><a href="https://www.archlinux.org/packages/extra/x86_64/htop/">htop</a></li>
<li><a href="https://wiki.archlinux.org/index.php/locate">mlocate</a></li>
<li><a href="https://wiki.archlinux.org/index.php/msmtp">msmtp</a></li>
<li><a href="https://wiki.archlinux.org/index.php/locale">locale</a></li>
<li><a href="https://wiki.archlinux.org/index.php/time">timedatectl</a></li>
<li><a href="https://github.com/xanmanning/alarm-fake-hwclock">fake-hwclock</a></li>
<li><a href="https://github.com/axkibe/lsyncd">lsyncd</a></li>
</ul>
<h1>Partitioning</h1>
<p>After a while I couldn't update any packages anymore cause my root partition was full. When looking at the partitions I noticed I had only 1.7G for my root partition available but I installed it on a 4G sd card.</p>
<p>Looking for a solution I discovered the base image I copied over is by default configured for 2G cards. You have to expand the file system yourself if you want to benefit the full amount of storage on your sd card.</p>
<p>Following this <a href="http://jan.alphadev.net/post/53594241659/growing-the-rpi-root-partition">tutorial</a> I found on the net I achieved to grow my root partition without extracting the sd card.</p>
<h1>RAM</h1>
<p>Using ssh I will only use terminal connections to the pi without any graphical interfaces though the HDMI socket. By default the memory of the pi is shared for those graphical stuff and normal os operations.</p>
<p>I decided to decrease the amount of memory for the gpu by setting those amounts into the /boot/config.txt file:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="nv">gpu_mem_512</span><span class="o">=</span><span class="m">48</span>
<span class="w"> </span><span class="nv">gpu_mem_256</span><span class="o">=</span><span class="m">48</span>
</code></pre></div>
<p>That way I could benefit of more memory for the stuff I will run on the pi.</p>
<h1>Communication</h1>
<p>To be online day and night I installed a chat client using <a href="https://wiki.archlinux.org/index.php/Screen_Irssi_Bitlbee">bitlbee and irssi</a>. When I am not connected to this chat terminal and I get some message on one of the connected channels a notification will be sent through my android phone using <a href="https://irssinotifier.appspot.com/">irssinotifier</a> so I could decide if it's important enough to connect using my phone, <a href="https://play.google.com/store/apps/details?id=com.sonelli.juicessh">juicessh</a> or spin up my laptop.</p>
<p>Be sure to check out all the nifty <a href="http://scripts.irssi.org/">scripts</a> which can increase the joy of using the irssi chat client.</p>
<h1>RAID</h1>
<p>For one of projects running at <a href="https://www.inuits.eu">inuits</a> I was asked to set up a software raid using <a href="http://en.wikipedia.org/wiki/Mdadm">mdadm</a>. Through the years I have collected a bunch of USB disks.</p>
<p>Combining those two facts I figured I could set up a raid using a little usb hub.</p>
<h2>Model B</h2>
<p>I am aware of the bottle neck this hub creates to the raid setup but since it's not for a production environment and I like to play around I doesn't care about it :)</p>
<p><img alt="raspberry" src="../../images/raspberry/raspberry.jpg"></p>
<h2>Model B+</h2>
<p>An updated model of the raspberry pi was launched, the <a href="http://www.raspberrypi.org/introducing-raspberry-pi-model-b-plus/">model b+</a> extended with to 4 USB ports in total. So finally I could gain benefit of my raid setup.</p>
<p>I reconfigured my whole RAID setup by using those 4 individual USB sockets instead of the hub I used before. And man what a difference! It runs a lot faster and is a lot more use full and efficient nowadays.</p>
<p>Since there are no case available yet for this model I looked on <a href="http://thingiverse.com">thingiverse</a> and found a closed <a href="http://www.thingiverse.com/thing:398399">case</a> which I printed out on the ultimaker2 robot we can use at the office. It's a neat case but the top isn't clicked in to the bottom part so I had to use tape to stick them together..</p>
<p><img alt="raspberry-b-open" src="../../images/raspberry/raspberry-b-open.jpg">
<img alt="raspberry-b-closed" src="../../images/raspberry/raspberry-b-closed.jpg"></p>
<h1>Configuration</h1>
<p>This whole setup is based on the well documented archlinux wiki <a href="https://wiki.archlinux.org/index.php/RAID">page</a></p>
<p>4 sticks of 2 GB were formatted to ext4:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="c1"># mkfs.ext4 /dev/sdXX -L NAME</span>
</code></pre></div>
<p>Next step was to identify the UUID's of those freshly created volumes so I could use those to initialize the actual raid setup.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>blkid
<span class="w"> </span>/dev/sde1:<span class="w"> </span><span class="nv">LABEL</span><span class="o">=</span><span class="s2">"three"</span><span class="w"> </span><span class="nv">UUID</span><span class="o">=</span><span class="s2">"39564f64-18ed-4f0b-a2d8-9b2d7c62032a"</span><span class="w"> </span><span class="nv">TYPE</span><span class="o">=</span><span class="s2">"ext4"</span><span class="w"> </span><span class="nv">PARTUUID</span><span class="o">=</span><span class="s2">"91f72d24-01"</span>
<span class="w"> </span>/dev/sdb1:<span class="w"> </span><span class="nv">LABEL</span><span class="o">=</span><span class="s2">"four"</span><span class="w"> </span><span class="nv">UUID</span><span class="o">=</span><span class="s2">"9b37dc7f-3f0c-44ba-846b-e9ba9efaa03a"</span><span class="w"> </span><span class="nv">TYPE</span><span class="o">=</span><span class="s2">"ext4"</span><span class="w"> </span><span class="nv">PARTUUID</span><span class="o">=</span><span class="s2">"688434a6-01"</span>
<span class="w"> </span>/dev/sdd1:<span class="w"> </span><span class="nv">LABEL</span><span class="o">=</span><span class="s2">"two"</span><span class="w"> </span><span class="nv">UUID</span><span class="o">=</span><span class="s2">"321d1d03-eb87-4129-83c7-ee1ce232d1c1"</span><span class="w"> </span><span class="nv">TYPE</span><span class="o">=</span><span class="s2">"ext4"</span><span class="w"> </span><span class="nv">PARTUUID</span><span class="o">=</span><span class="s2">"9852d7fa-01"</span>
<span class="w"> </span>/dev/sdc1:<span class="w"> </span><span class="nv">LABEL</span><span class="o">=</span><span class="s2">"one"</span><span class="w"> </span><span class="nv">UUID</span><span class="o">=</span><span class="s2">"d61e46bd-9a28-427e-9a85-94dc292463ec"</span><span class="w"> </span><span class="nv">TYPE</span><span class="o">=</span><span class="s2">"ext4"</span><span class="w"> </span><span class="nv">PARTUUID</span><span class="o">=</span><span class="s2">"00099342-01"</span>
</code></pre></div>
<p>Using the gathered UUID's I then created a raid5 using 3 active sticks and one as hot spare device:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>mdadm<span class="w"> </span>--create<span class="w"> </span>--verbose<span class="w"> </span>--level<span class="o">=</span><span class="m">5</span><span class="w"> </span>--metadata<span class="o">=</span><span class="m">1</span>.2<span class="w"> </span>--chunk<span class="o">=</span><span class="m">256</span><span class="w"> </span>--raid-devices<span class="o">=</span><span class="m">3</span><span class="w"> </span>/dev/md0<span class="w"> </span>/dev/disk/by-uuid/d61e46bd-9a28-427e-9a85-94dc292463ec<span class="w"> </span>/dev/disk/by-uuid/321d1d03-eb87-4129-83c7-ee1ce232d1c1<span class="w"> </span>/dev/disk/by-uuid/39564f64-18ed-4f0b-a2d8-9b2d7c62032a<span class="w"> </span>--spare-devices<span class="o">=</span><span class="m">1</span><span class="w"> </span>/dev/disk/by-uuid/9b37dc7f-3f0c-44ba-846b-e9ba9efaa03a
<span class="w"> </span>mdadm:<span class="w"> </span>layout<span class="w"> </span>defaults<span class="w"> </span>to<span class="w"> </span>left-symmetric
<span class="w"> </span>mdadm:<span class="w"> </span>/dev/disk/by-uuid/d61e46bd-9a28-427e-9a85-94dc292463ec<span class="w"> </span>appears<span class="w"> </span>to<span class="w"> </span>contain<span class="w"> </span>an<span class="w"> </span>ext2fs<span class="w"> </span>file<span class="w"> </span>system
<span class="w"> </span><span class="nv">size</span><span class="o">=</span>1982464K<span class="w"> </span><span class="nv">mtime</span><span class="o">=</span>Thu<span class="w"> </span>Jan<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">01</span>:00:00<span class="w"> </span><span class="m">1970</span>
<span class="w"> </span>mdadm:<span class="w"> </span>/dev/disk/by-uuid/321d1d03-eb87-4129-83c7-ee1ce232d1c1<span class="w"> </span>appears<span class="w"> </span>to<span class="w"> </span>contain<span class="w"> </span>an<span class="w"> </span>ext2fs<span class="w"> </span>file<span class="w"> </span>system
<span class="w"> </span><span class="nv">size</span><span class="o">=</span>2013184K<span class="w"> </span><span class="nv">mtime</span><span class="o">=</span>Thu<span class="w"> </span>Jan<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">01</span>:00:00<span class="w"> </span><span class="m">1970</span>
<span class="w"> </span>mdadm:<span class="w"> </span>/dev/disk/by-uuid/39564f64-18ed-4f0b-a2d8-9b2d7c62032a<span class="w"> </span>appears<span class="w"> </span>to<span class="w"> </span>contain<span class="w"> </span>an<span class="w"> </span>ext2fs<span class="w"> </span>file<span class="w"> </span>system
<span class="w"> </span><span class="nv">size</span><span class="o">=</span>2013184K<span class="w"> </span><span class="nv">mtime</span><span class="o">=</span>Thu<span class="w"> </span>Jan<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">01</span>:00:00<span class="w"> </span><span class="m">1970</span>
<span class="w"> </span>mdadm:<span class="w"> </span>/dev/disk/by-uuid/9b37dc7f-3f0c-44ba-846b-e9ba9efaa03a<span class="w"> </span>appears<span class="w"> </span>to<span class="w"> </span>contain<span class="w"> </span>an<span class="w"> </span>ext2fs<span class="w"> </span>file<span class="w"> </span>system
<span class="w"> </span><span class="nv">size</span><span class="o">=</span>1982464K<span class="w"> </span><span class="nv">mtime</span><span class="o">=</span>Thu<span class="w"> </span>Jan<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">01</span>:00:00<span class="w"> </span><span class="m">1970</span>
<span class="w"> </span>mdadm:<span class="w"> </span>size<span class="w"> </span><span class="nb">set</span><span class="w"> </span>to<span class="w"> </span>1981440K
<span class="w"> </span>mdadm:<span class="w"> </span>largest<span class="w"> </span>drive<span class="w"> </span><span class="o">(</span>/dev/disk/by-uuid/321d1d03-eb87-4129-83c7-ee1ce232d1c1<span class="o">)</span><span class="w"> </span>exceeds<span class="w"> </span>size<span class="w"> </span><span class="o">(</span>1981440K<span class="o">)</span><span class="w"> </span>by<span class="w"> </span>more<span class="w"> </span>than<span class="w"> </span><span class="m">1</span>%
<span class="w"> </span>Continue<span class="w"> </span>creating<span class="w"> </span>array?<span class="w"> </span>y
<span class="w"> </span>mdadm:<span class="w"> </span>array<span class="w"> </span>/dev/md0<span class="w"> </span>started.
</code></pre></div>
<p>You can see the progress of the creation by:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>cat<span class="w"> </span>/proc/mdstat
<span class="w"> </span>Personalities<span class="w"> </span>:<span class="w"> </span><span class="o">[</span>linear<span class="o">]</span><span class="w"> </span><span class="o">[</span>raid0<span class="o">]</span><span class="w"> </span><span class="o">[</span>raid1<span class="o">]</span><span class="w"> </span><span class="o">[</span>raid10<span class="o">]</span><span class="w"> </span><span class="o">[</span>raid6<span class="o">]</span><span class="w"> </span><span class="o">[</span>raid5<span class="o">]</span><span class="w"> </span><span class="o">[</span>raid4<span class="o">]</span><span class="w"> </span><span class="o">[</span>multipath<span class="o">]</span><span class="w"> </span><span class="o">[</span>faulty<span class="o">]</span>
<span class="w"> </span>md0<span class="w"> </span>:<span class="w"> </span>active<span class="w"> </span>raid5<span class="w"> </span>sde1<span class="o">[</span><span class="m">4</span><span class="o">]</span><span class="w"> </span>sdb1<span class="o">[</span><span class="m">3</span><span class="o">](</span>S<span class="o">)</span><span class="w"> </span>sdd1<span class="o">[</span><span class="m">1</span><span class="o">]</span><span class="w"> </span>sdc1<span class="o">[</span><span class="m">0</span><span class="o">]</span>
<span class="w"> </span><span class="m">3962880</span><span class="w"> </span>blocks<span class="w"> </span>super<span class="w"> </span><span class="m">1</span>.2<span class="w"> </span>level<span class="w"> </span><span class="m">5</span>,<span class="w"> </span>256k<span class="w"> </span>chunk,<span class="w"> </span>algorithm<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="o">[</span><span class="m">3</span>/2<span class="o">]</span><span class="w"> </span><span class="o">[</span>UU_<span class="o">]</span>
<span class="w"> </span><span class="o">[</span>>....................<span class="o">]</span><span class="w"> </span><span class="nv">recovery</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span>.8%<span class="w"> </span><span class="o">(</span><span class="m">36960</span>/1981440<span class="o">)</span><span class="w"> </span><span class="nv">finish</span><span class="o">=</span><span class="m">96</span>.5min<span class="w"> </span><span class="nv">speed</span><span class="o">=</span>335K/sec
<span class="w"> </span>unused<span class="w"> </span>devices:<span class="w"> </span><none>
</code></pre></div>
<p>Once the creation process has been done you can start by updating the configuration:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="c1"># mdadm --detail --scan >> /etc/mdadm.conf</span>
<span class="w"> </span><span class="c1"># mdadm --assemble --scan</span>
</code></pre></div>
<p>And we finally can create a file system on the raid itself:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="c1"># mkfs.ext4 -v -L NAME -m 0.5 -b 4096 -E stride=64,stripe-width=192 /dev/md0</span>
</code></pre></div>
<h2>Issues</h2>
<p>Since the raspberry is powered by the USB slot of my ISP's modem and they quite often restart their device for software updates the pi also rebooted from time to time. During such reboots I figured that the process of mounting the raid volume got stuck.</p>
<p>This because the hardware came up to slowly and the mounting process didn't recognized the usb sticks. So I wrote this script (/usr/bin/start-communication) which does the magic (after many try and error attempts).</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="c1">#!/bin/bash</span>
<span class="w"> </span><span class="c1">#</span>
<span class="w"> </span><span class="c1"># Script which mounts the RAID volume storage before starting an irssi screen session</span>
<span class="w"> </span><span class="c1"># Mount the storage RAID volume</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span>!<span class="w"> </span>df<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>NAME<span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"10 sec break..."</span><span class="p">;</span><span class="w"> </span>sleep<span class="w"> </span><span class="m">10</span>
<span class="w"> </span>sudo<span class="w"> </span>mount<span class="w"> </span>/dev/md0<span class="w"> </span>/NAME
<span class="w"> </span><span class="k">done</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"storage mounted"</span>
<span class="w"> </span><span class="c1"># Start irssi in a screen session as user 'X'</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span>!<span class="w"> </span>screen<span class="w"> </span>-list<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>irssi<span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="k">then</span><span class="w"> </span>sudo<span class="w"> </span>-u<span class="w"> </span>X<span class="w"> </span>/usr/bin/screen<span class="w"> </span>-dmS<span class="w"> </span>irssi<span class="w"> </span>irssi<span class="p">;</span><span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"irssi screen session started"</span>
</code></pre></div>
<p>This script is triggered by cron after every reboot.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="nv">@reboot</span><span class="w"> </span><span class="k">start</span><span class="o">-</span><span class="n">communication</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">startup</span><span class="p">.</span><span class="nf">log</span>
</code></pre></div>
<p>And the output is logged in the file /tmp/startup.log.</p>
<h1>Backup</h1>
<h2>image</h2>
<p>Once a week I overnight I have a cron job running which creates a compressed image file of the whole sd card of my running pi and pushes it to a mounted network share on my <a href="http://www.trustedreviews.com/iomega-tv-with-boxee_DVRs-and-Media-players-_review">boxee</a> iomega device.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="c1">#!/bin/bash</span>
<span class="w"> </span><span class="nv">DATE</span><span class="o">=</span><span class="k">$(</span>date<span class="w"> </span>+%d-%m-%Y<span class="k">)</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span>/usr/bin/mount<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>MOUNT<span class="p">;</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span>dd<span class="w"> </span><span class="k">if</span><span class="o">=</span>/dev/mmcblk0<span class="w"> </span><span class="p">|</span><span class="w"> </span>gzip<span class="w"> </span>-1<span class="w"> </span>-<span class="w"> </span><span class="p">|</span><span class="w"> </span>dd<span class="w"> </span><span class="nv">of</span><span class="o">=</span>MOUNT/backup-<span class="nv">$DATE</span>.img.gz
<span class="w"> </span><span class="k">fi</span>
</code></pre></div>
<p>That way I can easily pull this image onto a new sd card if there goes something wrong with the existing one without the need of reinstalling everything</p>
<h2>directory backup</h2>
<p>For some minor applications like irssi I use <a href="https://github.com/axkibe/lsyncd">lsyncd</a> to sync between different machines. I have one daemon running which is used to backup directories to a network share and another one to mirror between different machines through ssh.</p>
<h1>VPN</h1>
<p>One of the next steps will be a vpn setup based on this <a href="https://raymii.org/s/tutorials/IPSEC_L2TP_vpn_on_a_Raspberry_Pi_with_Arch_Linux.html">tutorial</a></p>
<h1>Wake on lan</h1>
<p>Once I got configured the VPN setup I will reconfigure my old laptop as being an <a href="https://visibilityspots.com/sms-server.html">sms-service</a>. Since I don't need this service being up all the time I will configure the wake on lan service on that laptop.</p>
<p>That way I can get him up from remote by logging in at my pi and sending the magical WOL packet to that laptop.</p>Octoprint2014-05-02T19:00:00+02:002017-05-10T00:00:00+02:00Jantag:visibilityspots.org,2014-05-02:/octoprint.html<p>At our <a href="https://inuits.eu">office</a> we have an <a href="https://www.ultimaker.com/pages/our-printers/ultimaker-2">ultimaker 2</a> installed, we have it now for some weeks and already printed out a bunch of stuff. It's cool to see, amazed by the technology.</p>
<p>Expect the manner to start a print job, you have to save your generated <a href="http://reprap.org/wiki/G-code">gcode</a> files onto an SD card, stick it into the ultimaker and use the rather simple menu from the tiny display to start the print job.</p>
<p>The time of a print job is rather long depending on the object you want to print, so many times you got up and went seeing if everything …</p><p>At our <a href="https://inuits.eu">office</a> we have an <a href="https://www.ultimaker.com/pages/our-printers/ultimaker-2">ultimaker 2</a> installed, we have it now for some weeks and already printed out a bunch of stuff. It's cool to see, amazed by the technology.</p>
<p>Expect the manner to start a print job, you have to save your generated <a href="http://reprap.org/wiki/G-code">gcode</a> files onto an SD card, stick it into the ultimaker and use the rather simple menu from the tiny display to start the print job.</p>
<p>The time of a print job is rather long depending on the object you want to print, so many times you got up and went seeing if everything went alright, so quickly distracted..</p>
<p>This sounds really crazy for such a piece of high tech technology and therefore I did a bit research on the web for a nice remote web interface.</p>
<p>It didn't cost me a lot of time/energy to find out about <a href="http://octoprint.org">octoprint</a></p>
<h1>Raspberry</h1>
<p>I first started by installing the octoprint manually on a raspbian distribution. But it took really long, I mean REALLY long so I downloaded the preconfigured <a href="https://github.com/guysoft/OctoPi">image</a>, maintained by <a href="http://osrc.dfm.io/guysoft">Guy Sheffer</a>, and installed it on a 4GB SD card.</p>
<p>I did some tweaks on the raspbian distribution, configured a for the pi user, disabled root login for the ssh service, updated the whole distribution and configured the wlan0 interface to connect wireless.</p>
<p>I struggled a bit with the configuration of the wireless part. Merely the pairwise parameter. I forgot to configure it to CCMP for the WPA2 settings of the wireless network. Resulting in the iwconfig output:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="n">unassociated</span><span class="w"> </span><span class="nl">Nickname</span><span class="p">:</span><span class="ss">"<WIFI@REALTEK>"</span>
</code></pre></div>
<p>Once I've setted up the configuration as you can see, network traffic came up on the wlan0 interface:
File: /etc/network/interfaces</p>
<div class="highlight"><pre><span></span><code> allow-hotplug wlan0
auto wlan0
iface wlan0 inet dhcp
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
</code></pre></div>
<p>File: /etc/wpa_supplicant/wpa_supplicant.conf</p>
<div class="highlight"><pre><span></span><code> ssid=""
scan_ssid=0
psk=""
proto=RSN
key_mgmt=WPA-PSK
pairwise=CCMP
auth_alg=OPEN
</code></pre></div>
<h1>Raspberry cam</h1>
<p>We also bought a camera <a href="http://www.raspberrypi.org/help/camera-module-setup/">module</a> which is discovered automatically by octoprint.</p>
<p>Once you've connected the hardware as described in the <a href="https://www.youtube.com/watch?v=GImeVqHQzsE">movie</a> and booted the raspberry the camera is immediately picked up by the octoprint service. So therefore you can't connect to the hardware when testing out <a href="http://www.raspberrypi.org/documentation/usage/camera/raspicam/raspistill.md">raspistill</a> and others to test the cam.</p>
<p>It took me some time to figure that out ;)</p>
<p>You should be able to catch the output of the mjpeg-streamer on port <a href="http://localhost:8080">8080</a> of the raspberry.</p>
<h1>Octoprint</h1>
<p>The octoprint interface is served on port <a href="http://localhost">80</a>. The account details are:</p>
<p>Where you can see the operational state of the printer if everything went well. The connection settings are:</p>
<ul>
<li>Serial Port: /dev/ttyACM0</li>
<li>Baudrate: 250000</li>
</ul>
<p>Normally the raspberry should connect using those settings automatically at startup to the ultimaker as you could configure in the settings page of octoprint.</p>
<h1>Usage</h1>
<p>You can look on for example <a href="http://software.ultimaker.com/">thingiverse</a> for source files of objects you want to print. (the .stl files). Which you have to slice using <a href="http://software.ultimaker.com/">cura</a>.</p>
<p>Be aware you set the machine settings of the cura to reprap (File -> Machine Settings -> GCodeFlavor -> RepRap (Marlin/Sprinter) ) so the octoprint engine can process your saved gcode files.</p>
<p>Once you've saved your gcode file you need to upload it using the octoprint webpage. After the upload is complete you can search your project in the Files section and click on the little printer icon on the right of the name.</p>
<p>You should see the temperature of the head is increasing to 220 degrees followed by the bed temperature of 70 degrees. Once the ultimaker is warmed up your print will start.</p>
<p>And you should see the progress of it in the octoprint interface.</p>
<h1>Pictures</h1>
<p><img alt="front" src="../../images/octoprint/front.jpg">
<img alt="side" src="../../images/octoprint/side.jpg">
<img alt="cam" src="../../images/octoprint/cam.jpg"></p>
<p>Happy printing!</p>Pakiti setup2014-04-25T19:00:00+02:002014-04-25T00:00:00+02:00Jantag:visibilityspots.org,2014-04-25:/pakiti.html<p>Nowadays it becomes more and more relevant knowing which version of a package is installed and having an overview of the packages infected by some bugs or security holes. That way you could see which servers are possibly vulnerable for those on the dark side.</p>
<p>That's where <a href="http://pakiti.sourceforge.net">pakiti</a> comes in a clean web based overview of your servers listing all packages vulnerable against the CVE in particular.</p>
<p><img alt="pakiti2" src="../../images/pakiti/pakiti2.png"></p>
<p>It's a client server setup where the client reports the version of the packages to the pakiti server. The server on his turn checks those versions against CVE to see if there are …</p><p>Nowadays it becomes more and more relevant knowing which version of a package is installed and having an overview of the packages infected by some bugs or security holes. That way you could see which servers are possibly vulnerable for those on the dark side.</p>
<p>That's where <a href="http://pakiti.sourceforge.net">pakiti</a> comes in a clean web based overview of your servers listing all packages vulnerable against the CVE in particular.</p>
<p><img alt="pakiti2" src="../../images/pakiti/pakiti2.png"></p>
<p>It's a client server setup where the client reports the version of the packages to the pakiti server. The server on his turn checks those versions against CVE to see if there are issues.</p>
<p>It is only a reporting tool, so pakiti will never install any package update, that still is and should be a controlled step which prevents unforeseen calls in the middle on the night when auto updates are breaking stuff.</p>
<h2>Pakiti server</h2>
<p>On the pakiti website they provide packages you could download and install out of the box. The pakiti-server needs a mysql database to store his data.</p>
<p>The whole installation process is pretty well documented in the documentation section of their homepage</p>
<p>You also need a webserver to deliver the pakiti web service.</p>
<p>A final step is the configuration from where pakiti would fetch the information about which packages are or are not a possible security risk for the server.</p>
<h2>Pakiti client</h2>
<p>On the servers you want to monitor the pakiti client package should be installed. Once a day this client will send a list with all packages installed on the system to the pakiti server through HTTP.</p>
<p>The configuration file could be found in /etc/pakiti2/pakiti-client.conf</p>
<h2>Puppet module</h2>
<p>Since I believe in automation I also wrote a <a href="https://github.com/visibilityspots/puppet-pakiti">puppet-pakiti</a> module which can be implemented in an existing puppet tree so you don't have to install and configure all your servers with the pakiti client.</p>
<p>Enjoy the overview, and happy updating ;)</p>Mkdocs documentation2014-04-20T17:00:00+02:002017-05-02T00:00:00+02:00Jantag:visibilityspots.org,2014-04-20:/mkdocs.html<p>To make our and other lives less painful writing documentation is a good start to decrease the level of frustration when working on a shared project.</p>
<p>It's a common feeling writing documentation isn't something we are all waiting for to do. In an effort to make it easier for all of us an automatically way of deployment can be managed by our good friend jenkins in combination with docker.</p>
<p>The details about this flow is been described on this page. After reading through this documentation section you should be aware of the general deployment idea so you can implement it …</p><p>To make our and other lives less painful writing documentation is a good start to decrease the level of frustration when working on a shared project.</p>
<p>It's a common feeling writing documentation isn't something we are all waiting for to do. In an effort to make it easier for all of us an automatically way of deployment can be managed by our good friend jenkins in combination with docker.</p>
<p>The details about this flow is been described on this page. After reading through this documentation section you should be aware of the general deployment idea so you can implement it yourself and start writing documentation without any hassle.</p>
<h2>Mark down</h2>
<p>The goal is that you write documentation using mark down in a git repository, that way you can easily write together with others on the same documentation in a structured and versionned manner.</p>
<p>By using mark down we can easily convert those md documents to whatever you want and gives us an easy <a href="https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet">syntax</a> to write documentation.</p>
<h2>Mkdocs</h2>
<p>Using <a href="http://mkdocs.org">mkdocs</a> a nice and easy manner has been found to generate a clean static html site based on the md files without much effort.</p>
<p>The installation is quite straight forward using python-pip:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>mkdocs
</code></pre></div>
<p>Once the installation process ended successfully you should be able to run the mkdocs engine:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdocs<span class="w"> </span><span class="nb">help</span>
<span class="w"> </span>mkdocs<span class="w"> </span><span class="o">[</span>help<span class="p">|</span>new<span class="p">|</span>build<span class="p">|</span>serve<span class="o">]</span><span class="w"> </span><span class="o">{</span>options<span class="o">}</span>
</code></pre></div>
<h3>Usage</h3>
<p>Starting a new project is easy as hell:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdocs<span class="w"> </span>new<span class="w"> </span>PROJECT-NAME
Creating<span class="w"> </span>project<span class="w"> </span>directory:<span class="w"> </span>PROJECT-NAME
Writing<span class="w"> </span>config<span class="w"> </span>file:<span class="w"> </span>PROJECT-NAME/mkdocs.yml
Writing<span class="w"> </span>initial<span class="w"> </span>docs:<span class="w"> </span>PROJECT-NAME/docs/index.md
</code></pre></div>
<p>the mkdocs structure is automatically generated as you can see in a brand new PROJECT-NAME directory:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>PROJECT-NAME
docs<span class="w"> </span>mkdocs.yml
</code></pre></div>
<p>As you could see the repository exists of a docs directory containing the md files with the actual content and a mkdocs.yml file which is used to generate the sites index and menus</p>
<h4>local preview</h4>
<p>The first thing you could do is to build a local preview of the html structure so you have a real time preview of your modifications:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>PROJECT-NAME
$<span class="w"> </span>mkdocs<span class="w"> </span>serve
Running<span class="w"> </span>at:<span class="w"> </span>http://127.0.0.1:8000/
Live<span class="w"> </span>reload<span class="w"> </span>enabled.
Hold<span class="w"> </span>ctrl+c<span class="w"> </span>to<span class="w"> </span>quit.
</code></pre></div>
<p>When the mkdocs engine started successfully you could surf through your browser to <a href="http://localhost:8000">localhost:8000</a> and start watching the preview of your documentation on your local machine.</p>
<p>You should see a site/ directory has been generated containing the static html structure based on the docs/ md files.</p>
<p>After editing the md files and saving your modifications they should appear immediately on your local preview when the mkdocs server command is running.</p>
<h4>mkdocs.yaml</h4>
<p>As mentioned a mkdocs.yml file manages the index and menus of the site:</p>
<div class="highlight"><pre><span></span><code><span class="nt">site_name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">My Docs</span>
<span class="nt">pages</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">index.md</span><span class="p p-Indicator">,</span><span class="w"> </span><span class="nv">Home</span><span class="p p-Indicator">]</span>
<span class="c1">#theme: readthedocs</span>
</code></pre></div>
<h4>images</h4>
<p>Using images is quite easy, add your jpg, png or whatever files into the docs/img/ directory and reference to them in your md file as follow:</p>
<div class="highlight"><pre><span></span><code>!<span class="o">[</span>reference<span class="w"> </span>name<span class="o">](</span>img/imagefile.png<span class="o">)</span>
</code></pre></div>
<h3>Migrate existing documentation</h3>
<p>Using <a href="http://johnmacfarlane.net/pandoc/">pandoc</a> we could convert the majority of source files to markdown:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pandoc<span class="w"> </span>source.txt<span class="w"> </span>-f<span class="w"> </span>textile<span class="w"> </span>-t<span class="w"> </span>markdown<span class="w"> </span>-o<span class="w"> </span>output.md
</code></pre></div>
<p>Be aware you should review the generated output cause the human intellect still cannot be fully replaced by bits and bytes..</p>
<h2>Automation</h2>
<p>Once you've written the documentation in markdown, checked locally the layout and ran through a spell checker you could push them to git repository.</p>
<p>A jenkins build flow could be triggered using a <a href="http://git-scm.com/book/en/Customizing-Git-Git-Hooks">post-receive</a> hooks.</p>
<p>This flow on his turn will orchestrates some jobs:</p>
<ul>
<li>build ( "package-doc" )</li>
<li>build ( "repository" )</li>
<li>build ( "deploy-package", packagename: "infra-doc", node: "webserver.domain.org")</li>
</ul>
<h3>Package-doc</h3>
<p>This job will use the git repository as a source to generate the html site/ directory by the mkdocs build command.</p>
<p>(tip: create a .gitignore file in the root of your git repo with <em>.~ </em>.swp and site/ so you don't upload swap files or you local generated site/ directory)</p>
<p>The nifty tool <a href="https://github.com/jordansissel/fpm">fpm</a> is used to generate an rpm package of that freshly created site/ directory to be deployed on hosting.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-f<span class="w"> </span>*.rpm<span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span>rm<span class="w"> </span>*.rpm
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>-d<span class="w"> </span><span class="s2">"site/"</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span>rm<span class="w"> </span>site/<span class="w"> </span>-rf
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span>mkdocs<span class="w"> </span>build
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>site/
<span class="w"> </span><span class="nv">RELEASE</span><span class="o">=</span><span class="sb">`</span>git<span class="w"> </span>rev-list<span class="w"> </span>--all<span class="w"> </span><span class="p">|</span><span class="w"> </span>wc<span class="w"> </span>-l<span class="sb">`</span>
<span class="w"> </span>fpm<span class="w"> </span>-s<span class="w"> </span>dir<span class="w"> </span>-t<span class="w"> </span>rpm<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--name<span class="w"> </span><span class="s2">"doc"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--version<span class="w"> </span><span class="s2">"1.0"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--iteration<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">RELEASE</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--architecture<span class="w"> </span>noarch<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--prefix<span class="w"> </span>/var/www/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--rpm-user<span class="w"> </span>apache<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--rpm-group<span class="w"> </span>apache<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--description<span class="w"> </span><span class="s1">'The html files for documentation'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--maintainer<span class="w"> </span><span class="s1">'Jenkins'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--epoch<span class="w"> </span><span class="s1">'1'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>.
<span class="w"> </span>mv<span class="w"> </span>*.rpm<span class="w"> </span>../
<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>..
<span class="w"> </span>rpm<span class="w"> </span>-qlp<span class="w"> </span>*.rpm
</code></pre></div>
<p>A brand new shiny rpm package artifact then could be archived so the next step in the flow could use it.</p>
<h3>Repository</h3>
<p>The rpm artifact of the package-doc job could then be used to deploy on your favorite repository service, from <a href="http://createrepo.baseurl.org/">createrepo</a>, <a href="http://pulpproject.org">pulp</a>, <a href="https://github.com/immobilienscout24/yum-repo-server">yum-repo-server</a>, <a href="https://github.com/dnbert/prm">prm</a> to <a href="http://packagecloud.io">packagecloud</a> so the next job can be triggered to install/update the package on your webserver</p>
<h3>Deploy-package</h3>
<p>Next you could configure a jenkins job which for examples logs in through ssh and installs the package you've pushed to your repository.</p>
<h3>Configuration management</h3>
<p>Instead of the deployment-package job you could also use a configuration management tool which does the installation/upgrade for you ;)</p>
<h2>Docker</h2>
<p>Instead of installing the tools on your local machine or your build server you could also opt for docker, there are a lot of preconfigured docker containers available on the internet or you could start making your own docker file relying on for example a centos official docker container and only mount your markdown documents into the container. That way you have more control over the environment and releases independent of the host system both by the ones who are writing the documentation as your build system..</p>
<h2>Useful links</h2>
<ul>
<li><a href="https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet">Adam-p</a></li>
<li><a href="https://help.github.com/articles/markdown-basics">Basics</a></li>
<li><a href="http://mkdocs.org">mkdocs</a></li>
<li><a href="http://johnmacfarlane.net/pandoc/">pandoc</a></li>
<li><a href="http://www.gnu.org/software/ispell/ispell.html">ispell</a></li>
</ul>CloudCollab Amsterdam #CCCEU132013-11-23T19:00:00+01:002013-11-23T00:00:00+01:00Jantag:visibilityspots.org,2013-11-23:/cloudcollab-amsterdam.html<p>Cloudstack, an item I had on my todo list with some lower priority against daily maintenance of our server park. But since attending <a class="reference external" href="https://twitter.com/ke4qqq">David Nalley's</a> talk on LinuxCon I shifted it up some places. Although I expected a real hands on session the talk he gave about a cloudstack environment for development was really intriguing and matched completely with what I had in mind.</p>
<p>Being fully convinced it fits in my idea of a fully automated development environment which meets to all the needs of developers to start writing code real quickly on machines similar to the production environment.</p>
<p>At …</p><p>Cloudstack, an item I had on my todo list with some lower priority against daily maintenance of our server park. But since attending <a class="reference external" href="https://twitter.com/ke4qqq">David Nalley's</a> talk on LinuxCon I shifted it up some places. Although I expected a real hands on session the talk he gave about a cloudstack environment for development was really intriguing and matched completely with what I had in mind.</p>
<p>Being fully convinced it fits in my idea of a fully automated development environment which meets to all the needs of developers to start writing code real quickly on machines similar to the production environment.</p>
<p>At that same conference LinuxCon I also attended a talk from <a class="reference external" href="https://twitter.com/jejb_">James Bottomley</a> about a container based cloud. I already did some stuff with lxc containers on my local machine hoping I could get it working one day so I could get rid of virtualbox and using containers for my puppet development instead. But I never thought about integrating containers in a cloud instead of the traditional hypervisors.</p>
<p>James gave a really inspiring talk about containers and passed his enthusiasm about it! Nevertheless it's a huge step to migrate from a traditional virtual based setup to a cloud based on containers. A huge challenge, but a challenge I can't wait to start on.</p>
<p>Some weeks after LinuxCon, CloudCollab took place, a chance I took with both hands when being asked to keep those days free. And so I drove to Amsterdam the evening before the event. I learned to read through the whole registration process and not let emotions and enthusiasm take it over so you forget or over read some major things. Thanks at those who fixed that the day before the event.</p>
<div class="section" id="hackathon-workshop-day">
<h2>Hackathon Workshop day</h2>
<p>I registered early at the <a class="reference external" href="http://www.beursvanberlage.nl/">Beurs van Berlage</a> where the conference is held cause I hate waiting queues. The opening talk was quite clear, we cloud/system admins must prevent the end users application developers and such to have frustrating moments where they banging their heads to the table.</p>
<p>With that said the workshops begun, I registered for the one-day cloudstack bootcamp by <a class="reference external" href="http://shapeblue.com/">shape blue</a>. After dealing with the exfat file system a friendly neighbor and changing some setting in the provisioned ova file for the xenserver I managed to go through the whole cycle of setting up domains, groups, accounts, networks, offering templates to finally getting up some vm's running and being able to access through ssh.</p>
<p>To do this we used the GUI, I'm looking forward to use the api they showed at the end of the day which looked far more my kinda usage then the GUI.</p>
<p>After the bootcamp my head was still dizzy but I took the opportunity to attend the <a class="reference external" href="http://www.meetup.com/ElasticSearch-NL/">elastic-search user group</a> meeting being held at booking.com. <a class="reference external" href="https://twitter.com/ralphm">Ralph Meijer</a> <a class="reference external" href="http://www.elasticsearch.org/blog/using-elasticsearch-and-logstash-to-serve-billions-of-searchable-events-for-customers/">spoke</a> about Logstach, Kibana and elasticsearch. Where <a class="reference external" href="https://github.com/mailgun/vulcan">vulcan</a> popped out for me as being interesting in combination with such a setup.</p>
</div>
<div class="section" id="conference-day-one">
<h2>Conference day one</h2>
<p>After being waked up 5 min early, I started the second day at CloudCollab by attending the Keynote of <a class="reference external" href="https://twitter.com/botchagalupe">John Willis</a> talking about the next frontier for devops. An entertaining talk but also very interesting topic. It seems like the history will repeat but now in networking area.</p>
<p>After the break I attended a talk about Devops, Killing of the Dinosaurs. Where all kind of culture troubles and people are compared to dinosaurs and how they achieved to kill them to get on.</p>
<p>I figured out I forgot my notebook at the keynote, luckily it was still where I left it, but I missed therefore the talks during that time. So I continued writing this post.</p>
<p>Next where the ignite talks, short talks where the slides are automatically flipped each 15 seconds. Unless you cheat and create duplicate slides off course. John Willis can talk like a machine, really really fast but still understandable.</p>
<p>Lunch being served stopped by some boots I started the afternoon by attending a talk about <a class="reference external" href="https://github.com/klarna/vagrant-cloudstack">vagrant-cloudstack</a>, which is really cool, finally I could perhaps using vagrant boxes exactly the same as a normal production server for development of puppet-modules without having to kickstart manually some boxes. This cloudstack virus is really getting me.</p>
<p><a class="reference external" href="http://www.slideshare.net/KrisBuytaert/the-future-of-sysadmin">The future of sysadmins</a>, finally I could attend a talk of our colleague <a class="reference external" href="https://twitter.com/krisbuytaert">Kris Buytaert</a>. Beside that fact I really was astonished that the way we are working at <a class="reference external" href="http://www.inuits.eu">Inuits</a> using automated pipelines, vagrant development, jenkins even pulp isn't yet commonly used. I couldn't believe I was like the only knowing some of the answers on Kris's questions. Obviously he would have nailed me when I didn't, but the only one? It's a mixed feeling being proud that we doing all those cool stuff, a bit disappointed not the majority of organizations are taking advantage of it.</p>
<p>After Kris's talk I went to a talk about monitoring a cloudstack environment. It felt like a sales talk for ca technologies own proprietary tool. Bit disappointing that it wasn't what I expected to be after reading the summary on lanyrd about the talk.</p>
<p>So I went for a coffee and bumped into some guys of the University Library of Cambridge at the Pinball machine in the dev room. Cool to see the story of their environment is quite the same as ours at the University Library of Ghent.</p>
<p>Being at the end of the day I followed a user panel about 4 organizations who implemented cloudstack for their business all with a different approach and goals. The one that popped out for me was <a class="reference external" href="http://www.greenqloud.com">Greenqloud</a> an Icelandic cloud provider running on 100% renewable energy (as everyone in Iceland), but which also does effort in other areas, like their hardware itself and the buildings their datacenters are deployed in.</p>
<p>After dinner we had a great time with the folks of <a class="reference external" href="http://shapeblue.com/">shape blue</a> and <a class="reference external" href="http://www.schubergphilis.com/">schuberg philis</a> at the pub. It's really fascinating to see such a dynamic atmosphere.</p>
</div>
<div class="section" id="conference-day-two">
<h2>Conference day two</h2>
<p>The last day of the conference started by checking out the hotel and attending the delayed keynote of <a class="reference external" href="https://twitter.com/markburgess_osl">Mark Burgess</a> about Uncertain Cloud Infrastructures. His book <a class="reference external" href="http://www.amazon.com/In-Search-Certainty-Information-Infrastructure-ebook/dp/B00ENEEWYO">In search of certainity</a> is added to my wish list for the upcoming holiday gifts.</p>
<p>Next talk I joined was about the <a class="reference external" href="https://github.com/apache/cloudstack/tree/master/plugins/file-systems/netapp">Netapp cloudstack plugin</a> which was really interesting, I hope I can get my hands on the beta version of the <a class="reference external" href="http://www.netapp.com/us/products/management-software/">VSC for cloudstack</a> software soon so I can start playing around with it on our test lab.</p>
<p>After being disappointed by a vendor talking about a topic which ended up in a sales talk I didn't had big expectations for the talk of <a class="reference external" href="http://www.linkedin.com/pub/mike-tutkowski/6/28/588">Mike Tutkowski</a> from Solid-fire about Guaranteed storage performance. But man how I was wrong. What a great talk. The guy really knew what he was talking about, explained how the storage area of cloudstack works and how they integrated it with their products. All vendor based sales talks should attend this talk and learn from it. That way more people could be becoming interested in your product only because of the clear and transparent explanation.</p>
<p>Because I'm looking for some scalable storage solution I attended the talk of <a class="reference external" href="https://twitter.com/widodh">Wido den Hollander</a> about <a class="reference external" href="http://www.ceph.com">ceph</a>. Wido is a passionate ceph lover who gave a crash course of ceph in 30 minutes. In that little time he really gave a clear overview of how ceph could be used together with cloudstack. Using little pizza box servers with one cpu and four disks you could easily manage your own ceph storage cluster.</p>
<p>After those 2 storage talks I came to this conclusion for myself that <a class="reference external" href="http://www.ceph.com">ceph</a> would be a great challenge if you want to keep control over your own storage soft- and hardware based besides the fact you also have to keep in mind about the physical space.</p>
<p>Another solution could be a solid-fire solution where you move the responsibility to a vendor. A great advantage of solid-file is that you can start with a small amount of data and grow your storage on a flexible and scalable manner to your own needs by just adding an extra node like the ceph solution and not like other vendors where you need to review the whole license contract.</p>
<p>I decided to attend the storage panel after those 2 talks being convinced that not only the cloud solution is important and changing the traditional ways of Virtualization but also storage is moving over to some more advanced flexible solutions.</p>
<p>Nevertheless I couldn't really hold my focus to the discussions being overwhelmed of the idea of the flexibility of those storage clusters being scalable, reliable and flexible volumes along on or more racks in multiple datacenters. I can only remember the statement of Wido: 'We still have storage problems. They are called NFS and iSCSI' because of my daydreams about storage clusters.</p>
<p>Being already 16hrs and a bit mental overwhelmed I was hesitating to leave already to home or attending the latest slots of talks. I decided to stay being interested about <a class="reference external" href="https://twitter.com/XenServerArmy">Tim Mackey</a>'s talk on the different hypervisors and how to choose between them to drive your cloud solution. He made a clear comparison between the different options. I hope I can catch his slides soon to share with you.</p>
<p>The closing note ended with a nice video about the conference was a great closing for a conference where I learned so many new technologies, options between the different solutions and inspiring people.</p>
<p>I want to thank hereby the people of <a class="reference external" href="http://www.schubergphilis.com/">Schuberg philis</a> for the organization!</p>
</div>
Git server2013-11-01T14:00:00+01:002013-11-01T00:00:00+01:00Jantag:visibilityspots.org,2013-11-01:/git-server.html<p>For some of my development projects I'm using git repositories because of the flexibility of it. But the initial beta phase I don't want to keep private until I created something working. Normally I use github.com repositories for them, a good service except you have to pay for private repositories.</p>
<p>So I searched the internet for private alternatives and installed <a class="reference external" href="http://www.gitlab.org">gitlab</a> on my CentOS 6 machine. It worked fine, but it was a bit of an overkill to manage about 10 repositories for only one user, myself. So I decided to migrate it back to the essence.</p>
<p>The essence …</p><p>For some of my development projects I'm using git repositories because of the flexibility of it. But the initial beta phase I don't want to keep private until I created something working. Normally I use github.com repositories for them, a good service except you have to pay for private repositories.</p>
<p>So I searched the internet for private alternatives and installed <a class="reference external" href="http://www.gitlab.org">gitlab</a> on my CentOS 6 machine. It worked fine, but it was a bit of an overkill to manage about 10 repositories for only one user, myself. So I decided to migrate it back to the essence.</p>
<p>The essence as: the command line git server with a nice web interface on top of it to have a quick overview of the changes made in which repositories.</p>
<p>I based my git server setup on the git-scm tutorial after reading the chapter about the <a class="reference external" href="http://git-scm.com/book/en/Git-on-the-Server">git-server</a>. It a clear and detailed explanation of the different steps to configure your own private git server.</p>
<p>Once the server was running and I could create new repositories, clone them and push to them from the outside I looked for a nice web frontend. My first choice was the <a class="reference external" href="https://git.wiki.kernel.org/index.php/Gitweb">git-web</a> interface with <a class="reference external" href="http://www.lighttpd.net/">lighttpd</a> as the backend web service. The installation of the <a class="reference external" href="http://git-scm.com/book/en/Git-on-the-Server-GitWeb">gitweb service</a> could also been found on git-scm.</p>
<p>For the lighttpd configuration I created a virtualhost pointing to the gitweb directory in /var/www/gitweb/.</p>
<p>/etc/lighttpd/vhosts.d/gitweb.conf:</p>
<pre class="literal-block">
$HTTP["url"] =~ "^/gitweb/" {
setenv.add-environment = ( "GITWEB_CONFIG" => "/etc/gitweb.conf" )
url.redirect += ( "^/gitweb$" => "/gitweb/" )
alias.url += ( "/gitweb/" => "/var/www/gitweb/" )
cgi.assign = ( ".cgi" => "" )
server.indexfiles = ( "gitweb.cgi" )
debug.log-request-header = "enable"
}
</pre>
<p>I used this interface for quite some time, but recently I found out about <a class="reference external" href="http://www.gitalist.com">gitalist</a>, a more modern approach to give and overview of your git repositories.</p>
<p>Gitalist is available as a perl-cpan module and could also been installed as such on a CentOS 6 server:</p>
<pre class="literal-block">
# cpan -i Gitalist
</pre>
<p>Until today I didn't got enough time to get it fully up and running, mainly because I already have something working so it's not that high on my priority list :)</p>
<p>Resources:</p>
<ul class="simple">
<li><a class="reference external" href="http://git-scm.com/">git-scm</a></li>
<li><a class="reference external" href="https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#Web_Interfaces">git web-interfaces</a></li>
</ul>
Dashing2013-10-31T21:30:00+01:002013-10-31T00:00:00+01:00Jantag:visibilityspots.org,2013-10-31:/dashing.html<p>Using multiple nice interface dashboards to get an overview of your services is a great thing. But navigating to them all separately could sometimes be rather pain full.</p>
<p>Therefore I looked for some central place to give a broad overview of all of them. During last year many passed through during my search on the internet. The 2 most interesting ones where <a class="reference external" href="http://fdietz.github.io/team_dashboard/">team dashboard</a> and <a class="reference external" href="http://shopify.github.io/dashing/">dashing</a>.</p>
<p>Team dashboard is a promising one which could gather extremely specific data and give those back in some nice graphics. That way you could create your own very specific dashboard with all graphics and …</p><p>Using multiple nice interface dashboards to get an overview of your services is a great thing. But navigating to them all separately could sometimes be rather pain full.</p>
<p>Therefore I looked for some central place to give a broad overview of all of them. During last year many passed through during my search on the internet. The 2 most interesting ones where <a class="reference external" href="http://fdietz.github.io/team_dashboard/">team dashboard</a> and <a class="reference external" href="http://shopify.github.io/dashing/">dashing</a>.</p>
<p>Team dashboard is a promising one which could gather extremely specific data and give those back in some nice graphics. That way you could create your own very specific dashboard with all graphics and measurements in the same theme/layout on one central page.</p>
<p>But I was looking for something more simpler and that's what I found with <a class="reference external" href="http://shopify.github.io/dashing/">dashing</a>. By using some custom jobs and views I gathered data from <a class="reference external" href="http://icinga.org/">icinga</a>, <a class="reference external" href="http://jenkins-ci.org/">jenkins</a>, <a class="reference external" href="http://theforeman.org">foreman</a> & <a class="reference external" href="http://bacula.org/">bacula</a>.</p>
<a class="reference external image-reference" href="images/dashing/rowOne.png"><img alt="Dashing first row" src="images/dashing/rowOne.png" /></a>
<a class="reference external image-reference" href="images/dashing/rowTwo.png"><img alt="Dashing second row" src="images/dashing/rowTwo.png" /></a>
<p>As you can see the square's are showing the total amount of checks from the different dashboard services, if there is one check failing the square of the service will change to a red blinking background. If everything is alright (as it should be) the square is green.</p>
<p>To achieve this I have implemented some checks I found on the internet and wrote some myself:</p>
<p>First row</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/roidelapluie/dashing-scripts/blob/master/jobs/icinga.rb">icinga-checks</a></li>
<li><a class="reference external" href="https://github.com/roidelapluie/dashing-scripts/blob/master/jobs/foreman.rb">foreman-overview</a></li>
<li><a class="reference external" href="https://github.com/roidelapluie/dashing-scripts/blob/master/jobs/jenkins.rb">jenkins-jobs</a></li>
<li><a class="reference external" href="https://github.com/visibilityspots/dashing-scripts#bacula-weberb">bacula-state</a></li>
</ul>
<p>The first three are using the simplemon widget available in the dashing-scripts repo from <a class="reference external" href="https://github.com/roidelapluie/dashing-scripts/">roidelaplui</a></p>
<p>Second row</p>
<ul class="simple">
<li><a class="reference external" href="https://gist.github.com/willjohnson/6313986">web-services</a></li>
<li><a class="reference external" href="https://gist.github.com/mavimo/6334816">jenkins-build-progress</a></li>
<li><a class="reference external" href="https://github.com/visibilityspots/dashing-scripts/blob/master/foursquare.rb">foursquare-checkins</a></li>
<li><a class="reference external" href="https://gist.github.com/sighmin/5628306">tomtom</a></li>
</ul>
<p>For the tomtom check the <a class="reference external" href="http://developer.tomtom.com/io-docs">api explorer</a> and <a class="reference external" href="http://www.satsig.net/maps/lat-long-finder.htm">lat-lon coordinates</a> which can be a real help to configure this check.</p>
<p>It's also real easy to configure a raspberry pi which you can connect to a screen using hdmi. Therefore I suggest <a class="reference external" href="http://www.screenlyapp.com/ose.html">screenly</a> which can iterate through a list of assets like web pages (your custom dashing screen ;), images and videos.</p>
<p>That way you could afford a cheap and brilliant monitor screen!</p>
<p>Keep an eye on it ;)</p>
LinuxCon Edinburgh2013-10-24T19:00:00+02:002013-10-24T00:00:00+02:00Jantag:visibilityspots.org,2013-10-24:/linuxcon-edinburgh.html<p>I got a great opportunity by attending LinuxCon in Edinburgh.</p>
<p>Will try to share my experiences there in this blogpost by listing the topics and people I found interesting so perhaps others could take also advantage of it.</p>
<div class="section" id="topics">
<h2>Topics</h2>
<p><a class="reference external" href="http://mesos.apache.org/">Apache mesos</a></p>
<p>a cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks.</p>
<p><a class="reference external" href="http://status.openstack.org/zuul/">Zuul</a></p>
<p>a pipeline oriented project gating and automation system.</p>
<p><a class="reference external" href="http://lttng.org/">LTTng</a></p>
<p>The LTTng project aims at providing highly efficient tracing tools for Linux. Its tracers help tracking down performance issues and debugging problems involving multiple concurrent processes and threads. Tracing across multiple systems is also …</p></div><p>I got a great opportunity by attending LinuxCon in Edinburgh.</p>
<p>Will try to share my experiences there in this blogpost by listing the topics and people I found interesting so perhaps others could take also advantage of it.</p>
<div class="section" id="topics">
<h2>Topics</h2>
<p><a class="reference external" href="http://mesos.apache.org/">Apache mesos</a></p>
<p>a cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks.</p>
<p><a class="reference external" href="http://status.openstack.org/zuul/">Zuul</a></p>
<p>a pipeline oriented project gating and automation system.</p>
<p><a class="reference external" href="http://lttng.org/">LTTng</a></p>
<p>The LTTng project aims at providing highly efficient tracing tools for Linux. Its tracers help tracking down performance issues and debugging problems involving multiple concurrent processes and threads. Tracing across multiple systems is also possible.</p>
<p><a class="reference external" href="http://www.opensaf.org/">SAF</a></p>
<p>OpenSAF is an open source project focused on Service Availability (SA) that goes beyond High Availability (HA) requirements. The goal of the OpenSAF project is to develop middle ware based on open and industry standard interfaces for applications requiring uninterrupted 24x7 service.</p>
<p><a class="reference external" href="http://itrevolution.com/books/phoenix-project-devops-book/">The phoenix project</a></p>
<p>A novel about IT, Devops and helping your business win.</p>
<p><a class="reference external" href="http://trac.buildbot.net/">Buildbot</a></p>
<p>Buildbot is a continuous integration system designed to automate the build/test cycle</p>
<p><a class="reference external" href="http://www.opendaylight.org/">Opendaylight</a></p>
<p>OpenDaylight's mission is to facilitate a community-led, industry-supported open source platform, including code and architecture, to accelerate adoption of Software-Defined Networking and Network Functions Virtualization.
hydrogen: <a class="reference external" href="http://www.opendaylight.org/news/2013/09/converge-network-digest-opendaylight-hydrogen-open-source-sdn">http://www.opendaylight.org/news/2013/09/converge-network-digest-opendaylight-hydrogen-open-source-sdn</a></p>
<p><a class="reference external" href="http://www.blackducksoftware.com/">Blackduck</a></p>
<p>Offers some services about open-source licensing</p>
</div>
<div class="section" id="interesting-people">
<h2>Interesting people:</h2>
<p><a class="reference external" href="https://twitter.com/bytebot">Colin Charles</a></p>
<p>Gave an entertaining talk about databases solutions in the cloud, with neat comparisons using relevant pro's and contra's between the different cloud based database providers without being boring just listening facts and numbers.</p>
<p><a class="reference external" href="https://twitter.com/jejb_">James Bottomley</a></p>
<p>A true container based cloud believer who inspired almost everyone in the room to use containers for their virtual appliances instead of the traditional virtual machines.</p>
<p><a class="reference external" href="https://twitter.com/ke4qqq">David Nalley</a></p>
<p>Gave an interesting presentation walk through how you could use cloudstack for a development environment.</p>
<p><a class="reference external" href="https://twitter.com/mikko">Mikko Hypponen</a></p>
<p>In times as today where the whole NSA topic is as hot as hell this guy talked about surveillance and got a great response from the audience talking as a keynote.</p>
</div>
CPAN rpm packages2013-10-05T14:00:00+02:002013-10-05T00:00:00+02:00Jantag:visibilityspots.org,2013-10-05:/cpan-rpm-packages.html<p>I went crazy from perl and the installation of their modules. For some icinga checks we need to install a few base perl packages using <a class="reference external" href="http://search.cpan.org/~miyagawa/App-cpanminus-1.7001/lib/App/cpanminus.pm">cpanminus</a>. It's taking a long time before the installation succeeds depending on the internet connection or server specifications.</p>
<p>Using a puppet exec to automate this installation is frustrating because the timeout is unpredictable and could take hours from time to time!</p>
<p>So I started to look for a way to package it into an rpm which I can distribute over our own yum repository.</p>
<p>The first software I got reviewed is <a class="reference external" href="http://perl.arix.com/cpan2rpm/">cpan2rpm</a>, it looked promising …</p><p>I went crazy from perl and the installation of their modules. For some icinga checks we need to install a few base perl packages using <a class="reference external" href="http://search.cpan.org/~miyagawa/App-cpanminus-1.7001/lib/App/cpanminus.pm">cpanminus</a>. It's taking a long time before the installation succeeds depending on the internet connection or server specifications.</p>
<p>Using a puppet exec to automate this installation is frustrating because the timeout is unpredictable and could take hours from time to time!</p>
<p>So I started to look for a way to package it into an rpm which I can distribute over our own yum repository.</p>
<p>The first software I got reviewed is <a class="reference external" href="http://perl.arix.com/cpan2rpm/">cpan2rpm</a>, it looked promising. You could give a text file containing the names of the modules to package.</p>
<p>That way I could use a git repo containing this file which triggers an automated <a class="reference external" href="http://jenkins-ci.org/">jenkins</a> job which creates the packages and uploads them to the repo.</p>
<p>Unfortunately it doesn't package the cpanminus module. So I had to look further.</p>
<p>Last week I got the solution by <a class="reference external" href="https://github.com/silug/cpanspec">cpanspec</a>, a piece of software I read about on <a class="reference external" href="http://nailingjelly.wordpress.com/2009/06/03/cpan-rpm-packaging/">nailingjelly</a> 's blogpost. And yes, I achieved to package it.</p>
<p>Installation & configuration of the required tools:</p>
<pre class="literal-block">
$ sudo yum install rpmdevtools perl perl-devel perl-Test-Base
$ sudo curl -L http://cpanmin.us | perl - --sudo App::cpanminus
$ sudo /usr/local/bin/cpanm CPAN::DistnameInfo
$ sudo yum install cpanspec
$ cd ~
$ rpmdev-setuptree
</pre>
<p>Create spec file and source rpm from a cpan module:</p>
<pre class="literal-block">
$ cpanspec --follow --srpm CPAN::Module --packager YOURNAME
</pre>
<p>Install the source rpm to create a package from it using the new generated spec file:</p>
<pre class="literal-block">
$ rpm -i name-of-module.src.rpm
</pre>
<p>You should see there is a SPEC file generated in the rpmbuild tree:</p>
<pre class="literal-block">
$ cd ~/rpmbuild/SPECS
$ vim cpan-module-name.spec
</pre>
<p>Finally give it a shot and build a fresh rpm package:</p>
<pre class="literal-block">
$ rpmbuild -ba cpan-module-name.spec
</pre>
<p>The first time trying to build App::cpanminus I had to add some missing file declarations to the spec file. Spawning the error:</p>
<pre class="literal-block">
RPM build errors:
Installed (but unpackaged) file(s) found:
/usr/bin/cpanm
/usr/share/man/man1/cpanm.1.gz
</pre>
<p>So I added the 2 unpacked files to the %files section:</p>
<pre class="literal-block">
%files
%defattr(-,root,root,-)
%doc Changes cpanfile LICENSE META.json README
%{perl_vendorlib}/*
%{_mandir}/man3/*
/usr/bin/cpanm
/usr/share/man/man1/cpanm.1.gz
</pre>
<p>Running the rpmbuild now resulted in a fresh rpm:</p>
<pre class="literal-block">
$ ls ../RPMS/noarch/
perl-App-cpanminus-1.7001-1.el6.noarch.rpm
</pre>
<p>I installed the rpm on a development system and successfully installed a perl module with the cpanm command afterwards:</p>
<pre class="literal-block">
$ yum localinstall name-of-the-module.rpm
</pre>
<p>So from now on our servers are hooked up with those create packages distributed by our own yum repository.</p>
<p>And the whole initialization process of a fresh server gained in time and therefore in efficiency in our environment this way!</p>
<p>Resources:</p>
<ul class="simple">
<li><a class="reference external" href="http://nailingjelly.wordpress.com/2009/06/03/cpan-rpm-packaging/">nailingjelly</a></li>
<li><a class="reference external" href="http://cpanspec.sourceforge.net/cpanspec.1.html">man</a> cpanspec</li>
<li>Centos.org <a class="reference external" href="http://wiki.centos.org/HowTos/RebuildSRPM">wiki</a></li>
</ul>
Taskwarrior2013-09-30T21:45:00+02:002013-09-30T00:00:00+02:00Jantag:visibilityspots.org,2013-09-30:/taskwarrior.html<p>I've used a lot's of tools to get a grip on my todo lists for work, for the scouting movement, for technical projects, household, etc. Started by using pen and paper, switched to a little notebook (which I still use for short-term tasks) to start using software to organize them.</p>
<p>I've used evernote, gtasks, tracks, github issues, gitlab issues, redmine tickets, in short plenty passed by only <a class="reference external" href="http://www.visibilityspots.com/tracks.html">tracks</a> survived. I still use it for my work related projects, everyday at 8:30AM I get my list of tasks for that day. That way I have some sort of control on …</p><p>I've used a lot's of tools to get a grip on my todo lists for work, for the scouting movement, for technical projects, household, etc. Started by using pen and paper, switched to a little notebook (which I still use for short-term tasks) to start using software to organize them.</p>
<p>I've used evernote, gtasks, tracks, github issues, gitlab issues, redmine tickets, in short plenty passed by only <a class="reference external" href="http://www.visibilityspots.com/tracks.html">tracks</a> survived. I still use it for my work related projects, everyday at 8:30AM I get my list of tasks for that day. That way I have some sort of control on my projects.</p>
<p>Nevertheless there was still some sort of missing feature, an integration with the other issue trackers I use like github and redmine for example. I dreamed of one central overview of all my tasks/issues/projects. And some weeks ago I just stumbled into the solution of that dream, <a class="reference external" href="http://taskwarrior.org">taskwarrior</a> will organize my life from now on.</p>
<p>It's a nifty command line based piece of software with all the features I needed, due dates, projects, tags, customized reports, etc. I completely get enthusiastic when finding out the <a class="reference external" href="https://github.com/ralphbean/bugwarrior">bugwarrior</a> module from Ralph Bean which let you to import tasks from many different services like github, redmine & trac.</p>
<p>So I started on this new project by adding a new task to my tracks instance: "Migrate to taskwarrior".</p>
<p>Installation of the <a class="reference external" href="http://taskwarrior.org/projects/taskwarrior/wiki/Download">task service</a></p>
<pre class="code bash literal-block">
<span class="c1"># yum install task</span>
</pre>
<p>By following the <a class="reference external" href="http://taskwarrior.org/projects/taskwarrior/wiki/30-second_Tutorial">30 sec tutorial</a> you get an idea of the basics, but for a full experience and howto I recommend reading the full <a class="reference external" href="http://taskwarrior.org/projects/taskwarrior/wiki/Tutorial">tutorial</a>. I created a dedicated user for managing my todo list on my CentOS 6.4 machine.</p>
<p>Configuration of the task service is done in the ~/.taskrc file where you can change the data & log files locations, setting a theme a other configuration parameters.</p>
<p>Installation of <a class="reference external" href="http://theunraveler.github.io/taskwarrior-web/">task-web</a>, a nice and clear frontend (make sure to use ruby 1.9.3, I had performance issues when using ruby 2.0.0):</p>
<pre class="code bash literal-block">
<span class="c1"># gem install taskwarrior-web thin
</span>$<span class="w"> </span>task-web<span class="w"> </span>-s<span class="w"> </span>thin<span class="w"> </span>-L<span class="w"> </span><span class="p">&</span>
</pre>
<p>I added the task-web.user & task-web.passwd parameters to my ~/.taskrc file for basic http authentication, and opted for the thin webserver rather than the default webrick when using the task-web frontend. Once you've stared the service your instance should be accessible on <a class="reference external" href="http://your.ip.of.the.server:5678">http://your.ip.of.the.server:5678</a> in your web browser. (make sure to open the port in your servers firewall)</p>
<p>You can choose your own port by adding the option -p XXX in your command (task-web -s thin -p XXX -L &). All the options are listed in the help menu (task-web --help).</p>
<p>Installation of <a class="reference external" href="https://github.com/ralphbean/bugwarrior">bugwarrior</a>:</p>
<p>As mentioned before the biggest advantage of using taskwarrior to me is the import feature of some several third party services. It's easy to install by using the <a class="reference external" href="http://www.pip-installer.org/en/latest/">pip installer</a>:</p>
<pre class="code bash literal-block">
$<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>bugwarrior
</pre>
<p>After that you can configure the ~/.bugwarriorrc file to your needs. After some struggling I got it working with the great help of the developer Ralph Bean.</p>
<p>Example of my ~/.bugwarriorrc file:</p>
<pre class="literal-block">
[general]
targets = github, redmine
log.level = INFO
log.file = /var/log/tasks/bugwarrior.log
bitly.api_user = USERNAME
bitly.api_key = API-KEY
multiprocessing = True
[notifications]
notifications = False
[github]
service = github
username = USERNAME
default_priority = M
login = USERNAME
passw = PASSWORD
[redmine]
service = redmine
url = https://redmine.url
key = REDMINE-API-KEY
user_id = NUMERIC-USER-ID
project_name = NAME-OF-THE-PROJECT-TASK-WILL-GET-ON-IMPORT
</pre>
<p>Once configured you can run the server and check the log's:</p>
<pre class="code bash literal-block">
$<span class="w"> </span>bugwarrior-pull<span class="w">
</span>$<span class="w"> </span>cat<span class="w"> </span>/var/log/tasks/bugwarrior.log<span class="w">
</span>$<span class="w"> </span>task<span class="w"> </span>list
</pre>
<p>Once you initialized the import you can create a cronjob for it:</p>
<pre class="literal-block">
$ crontab -e
# Bugwarrior import
30 5 * * * /usr/bin/bugwarrior-pull
</pre>
<p>That way every day at 5:30AM the tasks from 3Th party services will be imported.</p>
<p>The only feature I'm still missing is a 2 way synchronization. So I can edit the tasks in taskwarrior too, but that's something for utopia :)</p>
<p>Conky monitoring:</p>
<p>Is a already wrote about before I'm using <a class="reference external" href="http://www.visibilityspots.com/conky-colors.html">conky</a> as a dashboard together with my ratpoison setup. I already wrote a script to fetch my <a class="reference external" href="https://github.com/visibilityspots/scripts#conky-trackssh">tracks issues</a>. But now I need to fetch my task list from taskwarrior. So I created a custom task report configured in my ~/.taskrc file:</p>
<pre class="literal-block">
# Custom reports
report.conky.description=Conky report
report.conky.columns=project,description.truncated,depends.indicator,priority
report.conky.labels=Project,Desc,D,P
report.conky.sort=due+,project+,priority+
report.conky.filter=status:pending limit:page
</pre>
<p>Using a ssh connection you can then fetch the output from the command 'task conky' and parse it into a file using a bash script.</p>
<p>Because all my project definitions containing a hyphen I can parse them so I can grep titles and create new lines so I can parse them using the conky syntax.</p>
<!-- code ::bash
#!/bin/bash
ssh username@taskwarrior.server "task conky | head -7 | tail -4 | sed 's/^*[A-Z]*-[A-Z]*/&\n-/g' | sed -e 's/^- [ \t]*/ - /g' | sed 's/^/ /g' | head -4" -->
<p>I do still have 2 things I need to investigate time into:</p>
<p>Mail weekly tasks</p>
<p>Using <a class="reference external" href="http://pypi.python.org/pypi/taskreport/">taskreport</a> but I got some errors after installing using 'pip install taskreport':</p>
<pre class="literal-block">
$ taskreport
File "/usr/bin/taskreport", line 51
for key in ['userName', 'server', 'port']}
^
SyntaxError: invalid syntax
</pre>
<p>Installation of <a class="reference external" href="http://mirakel.azapps.de/taskwarrior.html">taskd</a> server (for synchronization with mirakel):</p>
<p>Until today the <a class="reference external" href="http://mirakel.azapps.de">mirakel</a> app always crashes when trying to sync after initialized with the created key.</p>
<pre class="code bash literal-block">
<span class="c1"># git clone git://tasktools.org/taskd.git
# wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
# rpm -Uvh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
# yum install cmake28
# yum install gnutls-devel
# yum install libuuid-devel
# cmake28 .
# make
# make install
</span><span class="w">
</span><span class="c1"># yum install gnutls-utils
# find and replace gnutls-certtool with certtool
# cd pki
# ./generate
</span><span class="w">
</span><span class="c1"># add_user.sh script</span>
</pre>
Upgrade to puppet 3.3.02013-09-20T19:00:00+02:002013-09-20T00:00:00+02:00Jantag:visibilityspots.org,2013-09-20:/puppet-3-upgrade.html<p>I finally got to the point where I upgraded a whole puppet infrastructure from puppet 2.6.x to the last stable version of puppet, <a class="reference external" href="http://docs.puppetlabs.com/puppet/3/reference/release_notes.html">3.3.0</a>. And after finding out the way to go it was surprisingly easy and no big issues came across.</p>
<p>One of the main reasons to upgrade was to start using the latest version of foreman, were we used 0.4, so we can start provisioning our own development vm's with some fancy cloud solution like for example <a class="reference external" href="http://cloudstack.apache.org/">cloudstack</a> using our production puppet tree.</p>
<p>Before the upgrade we had the puppet-client & server (2.6 …</p><p>I finally got to the point where I upgraded a whole puppet infrastructure from puppet 2.6.x to the last stable version of puppet, <a class="reference external" href="http://docs.puppetlabs.com/puppet/3/reference/release_notes.html">3.3.0</a>. And after finding out the way to go it was surprisingly easy and no big issues came across.</p>
<p>One of the main reasons to upgrade was to start using the latest version of foreman, were we used 0.4, so we can start provisioning our own development vm's with some fancy cloud solution like for example <a class="reference external" href="http://cloudstack.apache.org/">cloudstack</a> using our production puppet tree.</p>
<p>Before the upgrade we had the puppet-client & server (2.6.18), puppetdb (1.4), (ruby 1.8.7) and foreman (0.4.2) running on a CentOS 6.3 machine.</p>
<p>After upgrading we are running puppet-client & server (3.3.0) puppetdb (1.4), ruby (1.8.7) and foreman (1.2) all managed by puppet itself. (feels quite satisfying ;) )</p>
<p>The very fist time I started upgrading the puppet master, but instead of upgrading the puppet-server package from the yum puppetlabs repository I upgraded only the agent.</p>
<p>After I figured that out I could kill myself but ran out of time so needed to stop the process.</p>
<p>The second time I started totally in the wrong direction. I started with foreman, read about needing ruby 1.9.3. So I started looking for a CentOS 6.3 ruby 1.9.3 package.</p>
<p>Didn't find any started compiling it from source, but that came out on a total mess so I reverted my upgrade and postponed it for some days.</p>
<p>The final 3Th time I started in the right order. This order I will describe here:</p>
<p>(Before all those steps, make sure to disable puppet on your clients to have more control during the process)</p>
<div class="section" id="configuring-the-puppetlabs-repository">
<h2>Configuring the puppetlabs repository</h2>
<p>I like to install software from packages, so I started by configuring the <a class="reference external" href="http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html">puppetlabs</a> repository. I use a puppet-repo module for configuring repo's on our machines but you can quite easy install it from the command line.</p>
<p>This command is executed on a Cent0S 6.3 x86_64 machine:</p>
<pre class="literal-block">
# rpm -ivh http://yum.puppetlabs.com/el/6.0/products/x86_64/puppetlabs-release-6-7.noarch.rpm
</pre>
</div>
<div class="section" id="upgrading-the-puppetmaster">
<h2>Upgrading the puppetmaster</h2>
<p>So after shamelessly updated only the puppet package the first time, this time I did upgrade the puppet-server package without any issue. Be sure to read the <a class="reference external" href="http://docs.puppetlabs.com/guides/upgrading.html">docs</a> first about upgrading!</p>
<pre class="literal-block">
# yum update puppet-server
</pre>
<p>Once the puppetmaster is updated we can try our first puppet runs against the upgraded version.</p>
</div>
<div class="section" id="start-a-native-puppet-master-process-for-testing">
<h2>Start a native puppet master process for testing</h2>
<p>Before I get further in our upgrade process on passenger and stuff I wanted to know if the client is still able to do a puppet run without the passenger setup.</p>
<p>So I had to start the puppetmaster as a daemon, did a local puppet noop run on the master itself and stopped the puppetmaster daemon after I checked the run.</p>
<pre class="literal-block">
# puppet resource service puppetmaster ensure=running enable=true
# puppet agent --test --noop
# puppet resource service puppetmaster ensure=stopped enable=true
</pre>
</div>
<div class="section" id="upgrade-the-passenger-setup">
<h2>Upgrade the passenger setup</h2>
<p>We are using a passenger setup to have our puppet master in a scalable setup. Therefore we also needed to upgrade passenger on our puppetmaster and adopt the puppetmaster vhost to the upgraded environment.</p>
<p>To accomplish this I simply followed the <a class="reference external" href="http://docs.puppetlabs.com/guides/passenger.html">passenger</a> documentation of puppetlabs which was quite easy to follow.</p>
</div>
<div class="section" id="client">
<h2>Client</h2>
<p>Once the puppetmaster was upgraded I tested a puppet run with a still not updated client against the upgraded puppetmaster. It did the job except from sending reports. Since I planned to upgrade the clients too I did not invest time into this issue.</p>
<p>There fore I just upgraded the client itself where the puppetlabs repository already was enabled:</p>
<pre class="literal-block">
# yum update puppet
</pre>
</div>
<div class="section" id="issues">
<h2>Issues</h2>
<ul class="simple">
<li>403: authentication error</li>
</ul>
<p>By running my first 3.3.0 client vs the 3.3.0 master I got an authentication error 403 forbidden request. Did some research on the net, and found about an issue in the puppetmaster's <a class="reference external" href="http://projects.puppetlabs.com/issues/16765">auth.conf</a> file. Once I added this to the file:</p>
<pre class="literal-block">
# allow nodes to retrieve their own node definition
path ~ ^/node/([^/]+)$
method find
allow $1
</pre>
<p>The run did what I had to do configuring the server by using puppet!</p>
<ul class="simple">
<li>undefined method 'symbolize'</li>
</ul>
<p>On some clients I got this error message when trying to run puppet. On <a class="reference external" href="http://somethingsinistral.net/blog/the-angry-guide-to-puppet-3/">somethingsinistral.net</a> I found out it had to see with multiple puppet versions on your machine. By looking into the installed gems (make sure to check also possible rvm environments) and cleaned the ancient ones out I got the puppet run up and running again.</p>
<ul class="simple">
<li>icinga <a class="reference external" href="https://github.com/ripienaar/monitoring-scripts/issues/3">check_puppet</a></li>
</ul>
<p>We are using ripienaar's icinga check_puppet to monitor the puppet functionality. The became all red indicating puppet had too long not ran on the server. In the troubleshooting process I figured out the nagios user which is running the check over the NRPE protocol wasn't able to read the /var/lib/puppet/state/last_run_summary.yaml file. By checking permissions I found out the default settings of the /var/lib/puppet directory are 0750 when installing puppet.</p>
<p>Once I've changed them to 755 all check's became green again!</p>
</div>
<div class="section" id="foreman">
<h2>Foreman</h2>
<p>Once the puppet master was running fine again I also upgraded <a class="reference external" href="http://theforeman.org/manuals/1.2/index.html#3.3InstallFromPackages">theforeman</a> service running on the same machine as the puppetmaster. This went smoothly once I figured out the ruby and rake commands in the documentation must be replaced with ruby193-rake/ruby193-ruby when installed foreman from their repository.</p>
<p>Also do not forget to install foreman-mysql / foreman-sqlite etc when using those extra features.</p>
</div>
Command line printing & scanning2013-08-02T21:00:00+02:002013-08-02T00:00:00+02:00Jantag:visibilityspots.org,2013-08-02:/printing-scanning.html<p>Since I discovered the joy of using the ratpoison window manager I'm trying to do all tasks I need to perform on my system using the command line.</p>
<p>One of those frequently used tasks is printing out documents or scanning in files. Until today I used the software viewer of my documents to print and simple-scan to scan my files.</p>
<p>Nowadays I use the command line to perform those tasks. To print out documents I use the <a class="reference external" href="http://www.tldp.org/HOWTO/Printing-Usage-HOWTO-1.html">lp</a> command:</p>
<pre class="literal-block">
"Get the status off all printers on your system"
$ lpc status all
"Print the desired file to a specific printer"
$ lpr …</pre><p>Since I discovered the joy of using the ratpoison window manager I'm trying to do all tasks I need to perform on my system using the command line.</p>
<p>One of those frequently used tasks is printing out documents or scanning in files. Until today I used the software viewer of my documents to print and simple-scan to scan my files.</p>
<p>Nowadays I use the command line to perform those tasks. To print out documents I use the <a class="reference external" href="http://www.tldp.org/HOWTO/Printing-Usage-HOWTO-1.html">lp</a> command:</p>
<pre class="literal-block">
"Get the status off all printers on your system"
$ lpc status all
"Print the desired file to a specific printer"
$ lpr -P PRINTERNAME FILE/TO/PRINT.XX
"Show the printing queue"
$ lpq -P PRINTERNAME
"Cancel a specific print job using the queue id"
$ lprm ID
"Cancel all printing jobs"
$ lprm -
</pre>
<p>Those are the commands I regularly use to print my documents.</p>
<p>For scanning I use <a class="reference external" href="http://www.sane-project.org/man/scanimage.1.html">scanimage</a> from sane. There are too many options to explain so I just give hereby the one I use to scan A4 formatted files to pdf:</p>
<pre class="literal-block">
"List your scan devices"
$ scanimage -L
"Scan the image to a pdf file"
$ scanimage -p > fileName.pdf
</pre>
<p>Off course there are many ways to perform those tasks using the command line. Those are only the ones I use on my fedora machine. I'm always open for suggestions!</p>
CentOS 6.4 software raid & LVM2013-07-24T23:00:00+02:002013-07-24T00:00:00+02:00Jantag:visibilityspots.org,2013-07-24:/raid.html<p>Been asked to setup a software raid of 12TB on a minimal CentOS 6.4 installation with 5 disks of 3TB each. Never played with raid nor lvm before so the challenge was great!</p>
<p>I started by doing research about <a class="reference external" href="http://www.cyberciti.biz/tips/raid5-vs-raid-10-safety-performance.html">RAID</a>. Came to the conclusion that RAID 5 was the best option for our purpose. So kept looking for a way to implement a software raid and stumbled into <a class="reference external" href="http://linux.die.net/man/8/mdadm">mdadm</a>.</p>
<p>Using the information of <a class="reference external" href="http://richard.blog.kraya.co.uk/2012/04/27/3tb-hdd-raid5-centos-6-2/">Richard</a>'s and <a class="reference external" href="http://zackreed.me/articles/48-adding-an-extra-disk-to-an-mdadm-array">Zack Reed</a>'s blogs I easily setted up the raid array and created some lvm volumes on top of that.</p>
<p>Creating of 3TB …</p><p>Been asked to setup a software raid of 12TB on a minimal CentOS 6.4 installation with 5 disks of 3TB each. Never played with raid nor lvm before so the challenge was great!</p>
<p>I started by doing research about <a class="reference external" href="http://www.cyberciti.biz/tips/raid5-vs-raid-10-safety-performance.html">RAID</a>. Came to the conclusion that RAID 5 was the best option for our purpose. So kept looking for a way to implement a software raid and stumbled into <a class="reference external" href="http://linux.die.net/man/8/mdadm">mdadm</a>.</p>
<p>Using the information of <a class="reference external" href="http://richard.blog.kraya.co.uk/2012/04/27/3tb-hdd-raid5-centos-6-2/">Richard</a>'s and <a class="reference external" href="http://zackreed.me/articles/48-adding-an-extra-disk-to-an-mdadm-array">Zack Reed</a>'s blogs I easily setted up the raid array and created some lvm volumes on top of that.</p>
<p>Creating of 3TB partitions on the physical disks</p>
<pre class="literal-block">
# parted /dev/sdX
# (parted) mklabel gpt
# (parted) unit TB
# (parted) mkpart primary 0.00TB 3.00TB
# (parted) print
</pre>
<p>Creating the raid5 array with all the prepared disks</p>
<pre class="literal-block">
# mdadm --create /dev/md0 --level=raid5 --raid-devices=5 /dev/sdX# /dev/sdX# /dev/sdX# /dev/sdX# /dev/sd#
</pre>
<p>Viewing the state of creation of the new array</p>
<pre class="literal-block">
# watch cat /proc/mdstat
</pre>
<p>Once the array is successfully created you have to store it into the config file</p>
<pre class="literal-block">
# echo "DEVICE partitions" > /etc/mdadm.conf
# echo "MAILADDR root@localhost" >> /etc/mdadm.conf
# mdadm --detail --scan >> /etc/mdadm.conf
</pre>
<p>so we can start creating lvm volumes on top of them</p>
<pre class="literal-block">
# pvcreate /dev/md0
# vgcreate vg_NAME /dev/md0
# lvcreate --name lv_NAME -l 100%FREE vg_NAME
</pre>
<p>We now created a physical volume, a volume group and a logical volume which can easily be resized and moved on top of the raid5 setup</p>
<p>To start using the volume we finally have to create a file system on it and check if everything went alright</p>
<pre class="literal-block">
# mkfs.ext4 /dev/vg_NAME/lv_NAME
# fsck.ext4 -f /dev/vg_NAME/lv_NAME
</pre>
<p>After the succesfull file system chack I came to a working raid setup. Nevertheless I figured out that by rebooting the machine the raid array wasn't initializing as I expected. Instead of the md0 as configured the raid array was coming up as a read-only one named md127. I did some research and found a usefull topic on it on the redhat <a class="reference external" href="https://bugzilla.redhat.com/show_bug.cgi?id=606481">bugzilla</a> forum.</p>
<p>There I found out that this can be reactivated manually by stopping the read-only instance and reassambling the array based on the /etc/mdadm.conf file:</p>
<pre class="literal-block">
# mdadm --stop /dev/md172
# mdadm --assemble --scan
</pre>
<p>Although that's not really the best solution because you have to do this by every reboot of your system. So I looked a bit further and found out you can regenerate your initramfs image using your mdadm.conf file using dracut:</p>
<pre class="literal-block">
# mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.old
# dracut --mdadmconf --force /boot/initramfs-$(uname -r).img $(uname -r)
</pre>
<p>Once that images is build I successfully rebooted the system and the md0 came up without any problem.</p>
<p>The final step is to get the new logical volume mounted automatically at boot, therefore you have to add something in your /etc/fstab file:</p>
<pre class="literal-block">
# /dev/mapper/vg_NAME-lv_NAME /var/NAME ext4 defaults 1 1
</pre>
<p>Some useful commands</p>
<pre class="literal-block">
## Stop raid array
# mdadm --stop /dev/md0
## Start raid array
# mdadm --assemble --scan
</pre>
<p>Resources:</p>
<ul class="simple">
<li>tcpdump: <a class="reference external" href="http://www.tcpdump.com/kb/os/linux/removing-raid-devices.html">removing</a></li>
<li>tcpdump: <a class="reference external" href="http://www.tcpdump.com/kb/os/linux/starting-and-stopping-raid-arrays.html">restarting</a></li>
<li><a class="reference external" href="http://www.ducea.com/2009/03/08/mdadm-cheat-sheet/">cheat</a> sheet</li>
<li>raid <a class="reference external" href="https://wiki.xkyle.com/Mdadm#Pause_a_Verify_or_Rebuild">states</a></li>
<li><a class="reference external" href="http://www.howtoforge.com/how-to-create-a-raid1-setup-on-an-existing-centos-redhat-6.0-system">howtoforge</a> initramfs</li>
</ul>
Hubot, the github chat automated bot2013-06-03T19:00:00+02:002013-06-03T00:00:00+02:00Jantag:visibilityspots.org,2013-06-03:/hubot.html<p>Some weeks ago I was asked by a customer to implement a bot on an IRC channel. Did some research about this topic and stumbled on the github <a class="reference external" href="http://github.hubot.com">hubot</a>.</p>
<p>The installation on a dedicated server running CentOS 6, using the irc adapter isn't hard. By following those steps you can easily start your own bot on a specified IRC channel.</p>
<p>You need some pre installed packages:</p>
<pre class="literal-block">
# yum install openssl openssl-devel openssl-static crypto-utils expat expat-devel gcc-c++ git
</pre>
<p>After installed those pre requirements nodejs is the next service we need. You can install the newest version using rpm packages you can find …</p><p>Some weeks ago I was asked by a customer to implement a bot on an IRC channel. Did some research about this topic and stumbled on the github <a class="reference external" href="http://github.hubot.com">hubot</a>.</p>
<p>The installation on a dedicated server running CentOS 6, using the irc adapter isn't hard. By following those steps you can easily start your own bot on a specified IRC channel.</p>
<p>You need some pre installed packages:</p>
<pre class="literal-block">
# yum install openssl openssl-devel openssl-static crypto-utils expat expat-devel gcc-c++ git
</pre>
<p>After installed those pre requirements nodejs is the next service we need. You can install the newest version using rpm packages you can find on the internet. For example on my <a class="reference external" href="http://repository.visibilityspots.com/repoview">repo</a> or building it from source:</p>
<pre class="literal-block">
$ wget http://nodejs.org/dist/v0.8.17/node-v0.8.17.tar.gz
$ tar xf node-v0.8.17.tar.gz -C /usr/local/src && cd /usr/local/src/node-v0.8.17
$ ./configure && make && make install
</pre>
<p>As you can see in the output, npm is installed into the '/usr/local/bin/' directory. To get this working in bash you could add this directory into your $PATH environment</p>
<pre class="literal-block">
$ PATH=$PATH:/usr/local/bin/
</pre>
<p>So now we can use npm to install hubot and coffee-script:</p>
<pre class="literal-block">
$ npm install -g hubot coffee-script
</pre>
<p>You could now create your very own dedicated hubot by declaring the necessary files into your preferred path:</p>
<pre class="literal-block">
$ hubot -c /opt/hubot/
</pre>
<p>That way the core hubot you can use is installed in its own directory. We now have to install and configure the <a class="reference external" href="'https://github.com/github/hubot/wiki/Adapter:-IRC">irc-adapter</a>. Therefore you need to adapt the package.json file in your newly created hubot folder (/opt/hubot/) by inserting the hubot-irc dependency:</p>
<pre class="literal-block">
"dependencies": {
"hubot-irc": ">= 0.0.1",
"hubot": ">= 2.0.0",
...
}
</pre>
<p>Once that's done you can install the dependencies by using npm:</p>
<pre class="literal-block">
$ npm install
</pre>
<p>Last thing you have to do is configure the needed irc parameters. This can be done by exporting the environment parameters. I decided to use a file to accomplish this. In the /opt/hubot/ directory I created a hubot.env file containing the necessary parameters:</p>
<pre class="literal-block">
# IRC adapter parameters
export HUBOT_IRC_NICK="NAMEOFYOURBOT"
export HUBOT_IRC_ROOMS="#CHANNELONIRC"
export HUBOT_IRC_SERVER="irc.freenode.net"
export HUBOT_IRC_DEBUG="false"
export HUBOT_IRC_UNFLOOD="false"
export HUBOT_IRC_SERVER_FAKE_SSL"false"
export HUBOT_IRC_USESSL"false"
</pre>
<p>Most of those params are quite obvious, the unflood param configured to false prevented the hubot to crash when someone asked hubot: help ;)</p>
<p>After I tested this standard setup out I started to write a <a class="reference external" href="https://github.com/visibilityspots/puppet-hubot">puppet-hubot</a> module to automate those steps and configuration on a CentOS machine. Using a the <a class="reference external" href="https://github.com/visibilityspots/puppet-nodejs">puppet-nodejs</a> module which installs the <a class="reference external" href="http://nodejs.org">nodejs</a> rpm I packaged on my visibilityspots <a class="reference external" href="http://repository.visibilityspots.com/repoview">repo</a> the installation become easy peasy.</p>
<p>By using this puppet setup a hubot <a class="reference external" href="https://github.com/visibilityspots/scripts/blob/master/hubot">init</a> script is automatically deployed so a hubot init service can be used for starting, stopping, restarting and getting the status of the hubot service on your dedicated machine.</p>
<p>As you can see in the init script I use a hubot user to run the hubot. That way it's a bit more secure to run the hubot service on your server.</p>
<p>A 2nd script which is deployed using the puppet-hubot module is the <a class="reference external" href="https://github.com/visibilityspots/scripts/blob/master/hubot-plugin.sh">hubot-plugin.sh</a> script. By using this script you can automatically install a script from the hubot scripts <a class="reference external" href="http://hubot-script-catalog.herokuapp.com/">catalog</a>. If the author of the script uses the standard documentation rules the scripts will declare it self in your hubot-scripts.json file, declaring it's dependencies in the package.json file, if there are adding it's needed configuration parameters in plugins.env and restarting the hubot service.</p>
<p>If you notice a script which hasn't been documented the standard way, you can easily use pull requests, the author of the github hubot-scripts repository really takes it serious and merge those requests on a regular base.</p>
<p>Last but not least I also created a hubot instance using the <a class="reference external" href="https://github.com/markstory/hubot-xmpp">xmpp-adapter</a>. After some desperate debugging and failing I figured out that for the irc adapter it doesn't matter which nodejs version you installed. For the xmpp adapter on the other hand it only worked by installing nodejs v0.8.17 build from the sources and by never ever use npm update but npm install instead to install the npm dependencies.</p>
<p>It saves you a lot of time if you take that tip in memory :)</p>
<p>Enjoy using your hubot</p>
Chromium eid2013-05-29T19:00:00+02:002016-05-11T00:00:00+02:00Jantag:visibilityspots.org,2013-05-29:/chromium-eid.html<p>During this time of the year in Belgium most people needs to fill in their taxes forms.</p>
<p>Since a few years the Belgium government provided an electronic way to accomplish this task. Using a digital passport you can authenticate yourself.</p>
<p>I wanted to use this nice tool so I had to configure my local setup to have it all glued together on my linux machine.</p>
<p>The necessary steps I described in this post so other interested people can use their linux setups also to fill in the tax forms.</p>
<h1>Installation</h1>
<p>The mayor package to install on a fedora machine is …</p><p>During this time of the year in Belgium most people needs to fill in their taxes forms.</p>
<p>Since a few years the Belgium government provided an electronic way to accomplish this task. Using a digital passport you can authenticate yourself.</p>
<p>I wanted to use this nice tool so I had to configure my local setup to have it all glued together on my linux machine.</p>
<p>The necessary steps I described in this post so other interested people can use their linux setups also to fill in the tax forms.</p>
<h1>Installation</h1>
<p>The mayor package to install on a fedora machine is the <a href="https://code.google.com/p/eid-mw/">eid-mw</a> package:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>wget<span class="w"> </span>https://eid-mw.googlecode.com/files/eid-mw-4.0.0-0.925.fc16.x86_64.rpm
<span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>rpm<span class="w"> </span>-Uvh<span class="w"> </span>eid-mw-4.0.0-0.925.fc16.x86_64.rpm
</code></pre></div>
<p>If you are using archlinux on a dell latitude e6530 you can use the internal card reader by installing the drivers of the <a href="https://wiki.archlinux.org/index.php/Common_Access_Card">Common Access Card</a></p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>pacman<span class="w"> </span>-S<span class="w"> </span>pcsclite
<span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>vim<span class="w"> </span>/etc/opensc.conf
</code></pre></div>
<p>In the opensc.conf file you need to uncomment the setting <em>enable_pinpad = false</em> on two places before you enable the process at boot and run it:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>pcscd
<span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>start<span class="w"> </span>pcscd
</code></pre></div>
<p>So you could install the <a href="https://aur.archlinux.org/packages/eid-mw">eid-mw package</a> from the AUR repository</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>yaourt<span class="w"> </span>eid-mw
</code></pre></div>
<p>Once you've installed the <a href="https://code.google.com/p/eid-mw/">eid-mw</a> package on fedora and configured the pcscd service on archlinux you could install the firefox eid <a href="https://addons.mozilla.org/en-US/firefox/addon/belgium-eid/">addon</a> if you are using the firefox browser. Once that's accomplished you could test if it all works using the <a href="http://test.eid.belgium.be/">test page</a> provided by the Belgium government.</p>
<p>You can also use the <a href="https://code.google.com/p/eid-viewer/">eid-viewer</a> package, which provides you with a graphical piece of software so you can read out your passport, printing it out. Testing your pin code (if you forgot you're pincode you still have to go to your town services.</p>
<p>For fedora</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>wget<span class="w"> </span>https://eid-viewer.googlecode.com/files/eid-viewer-4.0.0-0.52.fc16.x86_64.rpm
<span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>rpm<span class="w"> </span>-Uvh<span class="w"> </span>eid-viewer-4.0.0-0.52.fc16.x86_64.rpm
</code></pre></div>
<p>For archlinux install the <a href="https://aur.archlinux.org/packages/eid-viewer/">eid-viewer package</a></p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>yaourt<span class="w"> </span>eid-viewer
</code></pre></div>
<p>Once the installation finished successfully you can run the software to view the information of your passport</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>eid-viewer
</code></pre></div>
<p>Still I'm not using firefox but the <a href="http://www.chromium.org">chromium-browser</a> to accomplish than I had to add the eid interface into the chromium security settings. I found an <a href="https://code.google.com/p/eid-mw/wiki/ChromeLinux">explanation</a> on google code and copied those steps into this post to be completed.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span><span class="c1"># only for fedora install nss-tools</span>
<span class="w"> </span>$<span class="w"> </span>sudo<span class="w"> </span>yum<span class="w"> </span>install<span class="w"> </span>nss-tools
<span class="w"> </span>$<span class="w"> </span>killall<span class="w"> </span>chromium-browser
<span class="w"> </span>or
<span class="w"> </span>$<span class="w"> </span>killall<span class="w"> </span>chromium
<span class="w"> </span>$<span class="w"> </span><span class="nb">cd</span>
<span class="w"> </span>$<span class="w"> </span>modutil<span class="w"> </span>-dbdir<span class="w"> </span>sql:.pki/nssdb/<span class="w"> </span>-add<span class="w"> </span><span class="s2">"Belgium eID"</span><span class="w"> </span>-libfile<span class="w"> </span>/usr/lib/libbeidpkcs11.so.0
<span class="w"> </span>$<span class="w"> </span>modutil<span class="w"> </span>-dbdir<span class="w"> </span>sql:.pki/nssdb/<span class="w"> </span>-list
</code></pre></div>
<p>So if you now start your chromium browser you could <a href="http://test.eid.belgium.be/">test</a> if it all works on your machine too :)</p>
<h1>Troubleshooting</h1>
<p>Since I only use this eid once a year and my system evolves in the meantime by installing rolling updates obviously issues arise..</p>
<div class="highlight"><pre><span></span><code> modutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
</code></pre></div>
<p>To solve his one I had to recreate my key database</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span>$<span class="w"> </span>mv<span class="w"> </span>.pki/nssdb<span class="w"> </span>.pki/nssdb.BAK
<span class="w"> </span>$<span class="w"> </span>mkdir<span class="w"> </span>.pki/nssdb
<span class="w"> </span>$<span class="w"> </span>certutil<span class="w"> </span>-N<span class="w"> </span>-d<span class="w"> </span>.pki/nssdb/
<span class="w"> </span>$<span class="w"> </span>modutil<span class="w"> </span>-create<span class="w"> </span>-dbdir<span class="w"> </span>.pki/nssdb/
<span class="w"> </span>$<span class="w"> </span>certutil<span class="w"> </span>-L<span class="w"> </span>-d<span class="w"> </span>.pki/nssdb/
<span class="w"> </span>$<span class="w"> </span>modutil<span class="w"> </span>-dbdir<span class="w"> </span>sql:.pki/nssdb/<span class="w"> </span>-add<span class="w"> </span><span class="s2">"Belgium eID"</span><span class="w"> </span>-libfile<span class="w"> </span>/usr/lib/libbeidpkcs11.so.0
<span class="w"> </span>$<span class="w"> </span>certutil<span class="w"> </span>-L<span class="w"> </span>-d<span class="w"> </span>.pki/nssdb/<span class="w"> </span>-h<span class="w"> </span><span class="s2">"all"</span>
<span class="w"> </span>$<span class="w"> </span>certutil<span class="w"> </span>-L<span class="w"> </span>-d<span class="w"> </span>.pki/nssdb/
</code></pre></div>
<p>When I achieved doeing so I could go ahead once again and fill in my taxes.</p>
<p>Resources:</p>
<ul>
<li><a href="http://eid.belgium.be/nl/je_eid_gebruiken/de_eid-middleware_installeren/linux/">eid-belgium</a></li>
</ul>Ratpoison window manager2013-05-22T19:00:00+02:002013-05-22T00:00:00+02:00Jantag:visibilityspots.org,2013-05-22:/ratpoison.html<p>My first steps in linux where on a ubuntu distribution, when you could order the ISO images on a cd-rom delivered by the post.</p>
<p>I liked it a lot and ever since I only used linux on my home based devices. Following the releases of Ubuntu. Starting at inuits I tried something else and installed CentOS desktop on my laptop. The idea behind this was to gain experience on the CentOS distributions.</p>
<p>Once I figured out that it didn't made sense since a laptop has other purposes then a server. By the time we got new machines I decided to …</p><p>My first steps in linux where on a ubuntu distribution, when you could order the ISO images on a cd-rom delivered by the post.</p>
<p>I liked it a lot and ever since I only used linux on my home based devices. Following the releases of Ubuntu. Starting at inuits I tried something else and installed CentOS desktop on my laptop. The idea behind this was to gain experience on the CentOS distributions.</p>
<p>Once I figured out that it didn't made sense since a laptop has other purposes then a server. By the time we got new machines I decided to install fedora on it. Nevertheless I don't like the gnome 3 unity layer. It's not that it's bad, But I just don't like it. So I started by installing <a class="reference external" href="http://mate-desktop.org">mate-desktop</a>.</p>
<p>By playing around and looking how other people are configuring and using their local machines a colleague pointed me to <a class="reference external" href="http://www.nongnu.org/ratpoison/">ratpoison</a>. Because I could install this window manager nicely next to the existing mate-desktop I gave it a try. Shameful I have to admit that when I first gave it a try I thought I did something wrong on the installation. That installation is not that hard on fedora, since they packaged it in their own repository:</p>
<pre class="literal-block">
$ sudo yum install ratpoison
</pre>
<p>Once installed you can logout and try to log in after changed your window manager. For me that first introduction was like I already mentioned a bit shameful. Nothing happened, I only saw a black background and couldn't do anything. It took me some time to figuring out that you had to use a keyboard pre configured strike to get started using the ratpoison functionality. And by default that is CTRL-T.</p>
<p>So if you for example tap in CTRL-T and then SHIFT-V you should see the ratpoison version disclaimer.</p>
<p>Once I figured that out I started to configure the whole environment for my needs. After some try and error I finally became in love with it. My screen movements are a lot faster by moving around screens, windows and applications trough my keyboard without physical moving my hands!</p>
<p>The configuration is done in the ~/.ratpoisonrc file:</p>
<pre class="literal-block">
# Ratpoison configuration
startup_message off
set winname class
# Desktop
set padding 0 0 0 93
exec conky -c ~/.conky/conkyrc
feh --bg-scale ~/path/to/background/picture.png
exec xscreensaver -nosplash
</pre>
<p>To begin I disable the startup message which only says what your keystroke is. Then I configure my desktop, setting a padding at the bottom of my screen so my <a class="reference external" href="http://www.visibilityspots.com/conky-colors.html">conky</a> setup is displayed smoothly on my screen. Starting the conky daemon, setting a background picture and starting the xscreensaver daemon.</p>
<pre class="literal-block">
# Startup programs
exec dropbox start
exec dropbox2
exec /home/Jan/.scripts/fnotify -s
exec /home/Jan/.scripts/ratcpi
exec /home/Jan/.scripts/detectPhone
</pre>
<p>The second part of my .ratpoisonrc file is the file with my startup scripts. To start my dropbox scripts as explained on a previous <a class="reference external" href="http://www.visibilityspots.com/dropbox.html">post</a>, the <a class="reference external" href="https://github.com/visibilityspots/scripts#fnotifysh">fnotify</a> script to display irssi notifications, <a class="reference external" href="https://github.com/jbaber/ratpoison_scripts/blob/master/Ratcpi/Ratcpi">ratcpi</a> for displaying battery notifications and <a class="reference external" href="https://github.com/vlachoudis/DetectPhone">detectPhone</a> which looks for my phone by bluetooth to decide to lock my laptop yes or no.</p>
<pre class="literal-block">
# Aliasses
alias showroot exec ratpoison -c $HOME/.rpfdump; ratpoison -c 'select -' -c only
alias unshowroot exec ratpoison -c "frestore at $HOME/.rpfdump"
alias showpadding set padding 0 0 0 93
alias showfullscreen set padding 0 0 0 0
alias term exec terminator
# Bindings
unbind n
unbind c
unbind s
unbind Q
## Software bindings
bind d exec chromium-browser
bind c exec terminator
bind C-c exec terminator
bind l exec xscreensaver-command -lock
bind C-s exec spotify
bind s exec synapse
## Move bindings
bind C-k delete
bind r restart
bind n nextscreen
bind C-n nextscreen
bind b showroot
bind B unshowroot
bind p showpadding
bind f showfullscreen
bind v hsplit
bind h vsplit
bind q only
</pre>
<p>This last part is about setting my key stroke bindings. Most of them are self explaining, all those keys need a pre hit of CTRL-T before called.</p>
Conky-colors2013-05-20T00:00:00+02:002013-05-20T00:00:00+02:00Jantag:visibilityspots.org,2013-05-20:/conky-colors.html<p>Back in the days I once wrote a <a class="reference external" href="http://www.visibilityspots.com/conky.html">blogpost</a> about a <a class="reference external" href="http://conky.sourceforge.net">conky</a> setup on a Ubuntu desktop. In the meantime I'm not using ubuntu anymore and kinda tweaked my whole conky setup. I switched to fedora 18 and using <a class="reference external" href="http://helmuthdu.deviantart.com/art/CONKY-COLORS-244793180">conky-colors</a> those days in front of the <a class="reference external" href="http://www.nongnu.org/ratpoison/">ratpoison</a> window manager.</p>
<p>This post will go trough all the steps I did to came to the actual result. When something isn't clear, or could be done on a more smoother/better way, please feel free to bug me about it!</p>
<p>Installing some required packages before actually compiling the conky-colors source:</p>
<pre class="literal-block">
$ sudo yum install …</pre><p>Back in the days I once wrote a <a class="reference external" href="http://www.visibilityspots.com/conky.html">blogpost</a> about a <a class="reference external" href="http://conky.sourceforge.net">conky</a> setup on a Ubuntu desktop. In the meantime I'm not using ubuntu anymore and kinda tweaked my whole conky setup. I switched to fedora 18 and using <a class="reference external" href="http://helmuthdu.deviantart.com/art/CONKY-COLORS-244793180">conky-colors</a> those days in front of the <a class="reference external" href="http://www.nongnu.org/ratpoison/">ratpoison</a> window manager.</p>
<p>This post will go trough all the steps I did to came to the actual result. When something isn't clear, or could be done on a more smoother/better way, please feel free to bug me about it!</p>
<p>Installing some required packages before actually compiling the conky-colors source:</p>
<pre class="literal-block">
$ sudo yum install hddtemp curl lm-sensors conky
</pre>
<p>Next thing is to scan your local machine for all available sensors (answering yes on all questions):</p>
<pre class="literal-block">
$ sudo sensors-detect
</pre>
<p>Once the pre requirements are successfully installed on your system we can start compiling the <a class="reference external" href="http://www.deviantart.com/download/244793180/conky_colors_by_helmuthdu-d41qrmk.zip?token=f47492fbd545e0ff90c83169ec04f24615aceb11&ts=1368796616">source</a> of conky colors:</p>
<pre class="literal-block">
$ cd into/unzipped/source/directory
$ make
$ sudo make install
</pre>
<p>Once that is done we can start by actually generating your desired conky setup. I based my setup on the SLIM theme and adopted it afterwards. Your custom weather code is based on the <a class="reference external" href="http://edg3.co.uk/snippets/weather-location-codes/">yahoo</a> service.</p>
<pre class="literal-block">
$ conky-colors --slim --w=1440 --h=900 --theme=white --weather=brxx0043
</pre>
<p>Two important files will be generated by this command. One will be located in your homedir and should be named like ~/.conkycolors/conkyrc. This is the config file we need to call when actually starting the conky daemon afterwards. Some configuration parameters to your specific layout can be adopted.</p>
<a class="reference external image-reference" href="images/conky-colors/initialSlim.png"><img alt="Initial conky slim view" src="images/conky-colors/initialSlim.png" /></a>
<p>I changed some of those params in the <a class="reference external" href="http://www.visibilityspots.com/documents/conky-colors/conkyrc">conkyrc</a> file, so the conky bar is located at the bottom om my screen (dell latitude E6530) with yellow colors and a border around it.</p>
<a class="reference external image-reference" href="images/conky-colors/initialCustom.png"><img alt="Changed color scheme to yellow view" src="images/conky-colors/initialCustom.png" /></a>
<pre class="literal-block">
own_window_type override
alignment bottom_left
gap_x 5
gap_y 5
minimum_size 1910 80
maximum_width 1910 80
draw_borders yes
color0 white
color1 CE5C00
color2 white
color3 CE5C00
</pre>
<p>The command to run the conky daemon:</p>
<pre class="literal-block">
$ conky -c ~/.conkyrc
$ CTRL-C to stop the job
# Move the conky job in the background
$ conky -c ~/.conkyrc &
</pre>
<p>We now should have a standard conky-colors theme running. But it wasn't sufficient for me. I kinda tweaked the whole setup and added some additional features:</p>
<a class="reference external image-reference" href="images/conky-colors/customizedSlim.png"><img alt="Customized view" src="images/conky-colors/customizedSlim.png" /></a>
<p>All those tweaks were added to the <a class="reference external" href="http://www.visibilityspots.com/documents/conky-colors/conkySlim-MainSection.lua">conkySlim.lua</a> file:</p>
<pre class="literal-block">
$ sudo vim /usr/share/conkycolors/scripts/conkySlim.lua
</pre>
<p>The fist tweak is the linux logo at the beginning of the conky screen (using the OpenLogos font):</p>
<pre class="literal-block">
--LOGO COLUMN
settings = {--VARIA
txt='Z',
x=50 , y=95 ,
txt_weight=0 , txt_size=100 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
font="OpenLogos"
};display_text(settings)
</pre>
<p>The second tweak is about the 2nd column in the conky output. I'll start with the uptime and update config which are quite straightforward:</p>
<pre class="literal-block">
--SECOND COLUMN
settings = {--UPTIME TITLE
txt=conky_parse("Uptime: "),
x=100 , y=20 ,
txt_weight=0 , txt_size=12 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
settings = {--UPTIME TITLE
txt=conky_parse("${uptime}"),
x=160 , y=20 ,
txt_weight=1 , txt_size=12 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
settings = {--UPDATES TITLE
txt=conky_parse("Updates: "),
x=100 , y=35 ,
txt_weight=0 , txt_size=12 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
};display_text(settings)
updates = conky_parse("${execi 360 yum -e0 -d0 check-update | wc -l}")
if updates > '9' then
color = theme
weight = '1'
message = 'available'
xAs = '175'
elseif updates > '0' then
color = theme
weight = '1'
message = 'available'
xAs = '180'
else
color = fgc
weight = '0'
message = ''
xAs = '175'
end
settings = {--# UPDATES
txt=updates,
x=160 , y=35 ,
txt_weight=weight , txt_size=12 ,
txt_fg_colour=color , txt_fg_alpha=fga ,
};display_text(settings)
settings = {--UPDATES MESSAGE
txt=message,
x=xAs , y=35 ,
txt_weight=0 , txt_size=12 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
};display_text(settings)
</pre>
<p>Following with my own <a class="reference external" href="http://www.irssi.org">irssi</a> status based on an html file which is generated on the server where the irssi daemon is running using the irssi script <a class="reference external" href="http://scripts.irssi.org/scripts/away2web.pl">away2web</a>. The username and password in this example are needed for basic htaccess authentication:</p>
<pre class="literal-block">
irssiState= conky_parse("${exec curl --user USERNAME:PASSWORD https://URLTOGENERATEDAWAY2WEBFILE/status.html -k -s | head -1}")
if irssiState == '1' then
color = theme
message = ''
state = 'Online'
else
color = theme
message = conky_parse("${exec curl --user USERNAME:PASSWORD https://URLTOGENERATEDAWAY2WEBFILE/status.html -k -s | tail -1}")
state = 'Offline '
end
settings = {--IRSSI TITLE
txt='Irssi:',
x=100 , y=51 ,
txt_weight=0 , txt_size=12 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
settings = {--IRSSI STATE
txt=state,
x=160 , y=51 ,
txt_weight=1 , txt_size=12 ,
txt_fg_colour=color , txt_fg_alpha=fga ,
};display_text(settings)
settings = {--IRSSI MESSAGE
txt=message,
x=207 , y=51 ,
txt_weight=0 , txt_size=10 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
</pre>
<p>And as last informational message in this column a minimal overview of your infrastructure based on an <a class="reference external" href="http://www.icinga.org">icinga</a> instance based on a <a class="reference external" href="https://github.com/visibilityspots/scripts#conky-icingash">conky-icinga</a> bash script:</p>
<pre class="literal-block">
--ICINGA STATE
IcingaState=conky_parse("${execpi 53 PATH/TO/conky-icinga.sh}")
if IcingaState == 'OK' then
color = fgc
elseif IcingaState == 'WARN' then
color = fgc
else
color = fgc
end
settings = {--ICINGA TITLE
txt='Icinga:',
x=100 , y=80 ,
txt_weight=0 , txt_size=12 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
};display_text(settings)
settings = {--ICINGA STATE
txt=IcingaState,
x=160 , y=80 ,
txt_weight=1 , txt_size=12 ,
txt_fg_colour=color , txt_fg_alpha=fga ,
};display_text(settings)
</pre>
<p>As you can see in the middle section I added a counter for incoming mails based on my maildir folders:</p>
<pre class="literal-block">
settings = {--MAILS
txt=conky_parse("Inuits: ${new_mails PATH/TO/MAILDIR}"),
x=(w/2)-160 , y=65 ,
txt_weight=0 , txt_size=12 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
};display_text(settings)
</pre>
<p>Depending if the spotify service is running conky will display the 'Now playing song - artist':</p>
<pre class="literal-block">
settings = {--SPOTIFY MUSIC SYMBOL
txt=conky_parse("${if_running spotify}z${endif}"),
x=(w/2)+60 , y=83 ,
txt_weight=0 , txt_size=10 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
font="musicelements"
};display_text(settings)
settings = {--SPOTIFY
txt=conky_parse("${if_running spotify}${exec sudo spotify-nowplaying}${endif}"),
x=(w/2)+67 , y=83 ,
txt_weight=0 , txt_size=10 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
</pre>
<p>The same counts for <a class="reference external" href="http://cmus.sourceforge.net/">cmus</a> a command line music player which will show 'Now playing song - artist when active' using a very basic <a class="reference external" href="https://github.com/visibilityspots/scripts#cmussh">script</a></p>
<pre class="literal-block">
settings = {--CMUS MUSIC SYMBOL
txt=conky_parse("${if_running cmus}z${endif}"),
x=(w/2)+60 , y=83 ,
txt_weight=0 , txt_size=10 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
font="musicelements"
};display_text(settings)
settings = {--CMUS
txt=conky_parse("${if_running cmus}${execi 2 ~/PATH/TO/cmus.sh}${endif}"),
x=(w/2)+67 , y=83 ,
txt_weight=0 , txt_size=10 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
</pre>
<p>4 cpu's will be used to draw the CPU graphics and showing the load of the local machine:</p>
<pre class="literal-block">
settings = {--CPU GRAPH CPU1
value=tonumber(conky_parse("${cpu cpu1}")),
value_max=100 ,
x=xp , y=yp ,
graph_radius=22 ,
graph_thickness=5 ,
graph_start_angle=180 ,
graph_unit_angle=2.7 , graph_unit_thickness=2.7 ,
graph_bg_colour=bgc , graph_bg_alpha=bga ,
graph_fg_colour=theme , graph_fg_alpha=fga ,
hand_fg_colour=theme , hand_fg_alpha=0.0 ,
txt_radius=35 ,
txt_weight=1 , txt_size=8.0 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
graduation_radius=28 ,
graduation_thickness=0 , graduation_mark_thickness=1 ,
graduation_unit_angle=27 ,
graduation_fg_colour=theme , graduation_fg_alpha=0.3 ,
caption='CPU' ,
caption_weight=1 , caption_size=10.0 ,
caption_fg_colour=fgc , caption_fg_alpha=fga ,
};draw_gauge_ring(settings)
settings = {--CPU GRAPH CPU2
value=tonumber(conky_parse("${cpu cpu2}")) ,
value_max=100 ,
x=xp , y=yp ,
graph_radius=17 ,
graph_thickness=5 ,
graph_start_angle=180 ,
graph_unit_angle=2.7 , graph_unit_thickness=2.7 ,
graph_bg_colour=bgc , graph_bg_alpha=bga ,
graph_fg_colour=theme , graph_fg_alpha=fga ,
hand_fg_colour=theme , hand_fg_alpha=0.0 ,
txt_radius=9 ,
txt_weight=1 , txt_size=8.0 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
graduation_radius=28 ,
graduation_thickness=0 , graduation_mark_thickness=1 ,
graduation_unit_angle=27 ,
graduation_fg_colour=theme , graduation_fg_alpha=0.3 ,
caption='' ,
caption_weight=1 , caption_size=10.0 ,
caption_fg_colour=fgc , caption_fg_alpha=fga ,
};draw_gauge_ring(settings)
settings = {--CPU GRAPH CPU3
value=tonumber(conky_parse("${cpu cpu3}")) ,
value_max=100 ,
x=xp , y=yp ,
graph_radius=17 ,
graph_thickness=5 ,
graph_start_angle=180 ,
graph_unit_angle=2.7 , graph_unit_thickness=2.7 ,
graph_bg_colour=bgc , graph_bg_alpha=bga ,
graph_fg_colour=theme , graph_fg_alpha=fga ,
hand_fg_colour=theme , hand_fg_alpha=0.0 ,
txt_radius=0 ,
txt_weight=1 , txt_size=8.0 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
graduation_radius=28 ,
graduation_thickness=0 , graduation_mark_thickness=1 ,
graduation_unit_angle=27 ,
graduation_fg_colour=theme , graduation_fg_alpha=0.3 ,
caption='' ,
caption_weight=1 , caption_size=10.0 ,
caption_fg_colour=fgc , caption_fg_alpha=fga ,
};draw_gauge_ring(settings)
settings = {--CPU GRAPH CPU4
value=tonumber(conky_parse("${cpu cpu4}")) ,
value_max=100 ,
x=xp , y=yp ,
graph_radius=17 ,
graph_thickness=5 ,
graph_start_angle=180 ,
graph_unit_angle=2.7 , graph_unit_thickness=2.7 ,
graph_bg_colour=bgc , graph_bg_alpha=bga ,
graph_fg_colour=theme , graph_fg_alpha=fga ,
hand_fg_colour=theme , hand_fg_alpha=0.0 ,
txt_radius=-9 ,
txt_weight=1 , txt_size=8.0 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
graduation_radius=28 ,
graduation_thickness=0 , graduation_mark_thickness=1 ,
graduation_unit_angle=27 ,
graduation_fg_colour=theme , graduation_fg_alpha=0.3 ,
caption='' ,
caption_weight=1 , caption_size=10.0 ,
caption_fg_colour=fgc , caption_fg_alpha=fga ,
};draw_gauge_ring(settings)
settings = {--LOAD
txt=conky_parse("${loadavg}"),
x=xp+10 , y=yp+38,
txt_weight=0 , txt_size=10 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
</pre>
<p>I also added a additional graph for the temperature based on acpi:</p>
<pre class="literal-block">
settings = {--TEMP GRAPH
value=tonumber(conky_parse("${acpitemp}")),
value_max=100 ,
x=xp , y=yp ,
graph_radius=22 ,
graph_thickness=5 ,
graph_start_angle=180 ,
graph_unit_angle=2.7 , graph_unit_thickness=2.7 ,
graph_bg_colour=bgc , graph_bg_alpha=bga ,
graph_fg_colour=theme , graph_fg_alpha=fga ,
hand_fg_colour=theme , hand_fg_alpha=0.0 ,
txt_radius=0 ,
txt_weight=1 , txt_size=8.0 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
graduation_radius=22 ,
graduation_thickness=4 , graduation_mark_thickness=2 ,
graduation_unit_angle=27 ,
graduation_fg_colour=theme , graduation_fg_alpha=0.5 ,
caption='TEMP' ,
caption_weight=1 , caption_size=10.0 ,
caption_fg_colour=fgc , caption_fg_alpha=fga ,
};draw_gauge_ring(settings)
</pre>
<p>I'm moving around a lot, connecting to wifi or wired depending on location. To let conky graph the right interface I wrote a wrapper around that:</p>
<pre class="literal-block">
iface = conky_parse("${exec ip n | awk {'print $3'} | head -1}")
if iface == 'em1' then
ifaceCaption = 'EM1'
else
ifaceCaption = 'WLAN0'
end
settings = {--NETWORK GRAPH UP
value=tonumber(conky_parse("${upspeedf " .. iface .. "}")),
value_max=100 ,
x=xp , y=yp ,
graph_radius=17 ,
graph_thickness=5 ,
graph_start_angle=180 ,
graph_unit_angle=2.7 , graph_unit_thickness=2.7 ,
graph_bg_colour=bgc , graph_bg_alpha=bga ,
graph_fg_colour=theme , graph_fg_alpha=fga ,
hand_fg_colour=theme , hand_fg_alpha=0.0 ,
txt_radius=0 ,
txt_weight=1 , txt_size=8.0 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
graduation_radius=28 ,
graduation_thickness=0 , graduation_mark_thickness=1 ,
graduation_unit_angle=27 ,
graduation_fg_colour=theme , graduation_fg_alpha=0.3 ,
caption='' ,
caption_weight=1 , caption_size=10.0 ,
caption_fg_colour=fgc , caption_fg_alpha=fga ,
};draw_gauge_ring(settings)
settings = {--NETWORK GRAPH DOWN
value=tonumber(conky_parse("${downspeedf " .. iface .. "}")),
value_max=100 ,
x=xp , y=yp ,
graph_radius=22 ,
graph_thickness=5 ,
graph_start_angle=180 ,
graph_unit_angle=2.7 , graph_unit_thickness=2.7 ,
graph_bg_colour=bgc , graph_bg_alpha=bga ,
graph_fg_colour=theme , graph_fg_alpha=fga ,
hand_fg_colour=theme , hand_fg_alpha=0.0 ,
txt_radius=35 ,
txt_weight=1 , txt_size=8.0 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
graduation_radius=28 ,
graduation_thickness=0 , graduation_mark_thickness=1 ,
graduation_unit_angle=27 ,
graduation_fg_colour=theme , graduation_fg_alpha=0.3 ,
caption=ifaceCaption ,
caption_weight=1 , caption_size=10.0 ,
caption_fg_colour=fgc , caption_fg_alpha=fga ,
};draw_gauge_ring(settings)
</pre>
<p>Depending on those locations I get other ip addresses on other networks and therefore other SMTP services. To tackle those smtp service I wrote a <a class="reference external" href="https://github.com/visibilityspots/scripts#setsmtpsh">setsmtp</a> script which will be called from within my conky setup based on the ip addresses:</p>
<pre class="literal-block">
if iface =='em1' then
ip = conky_parse("${addr em1}")
if ip == 'IP AT WORK PLACE ONE' then
conky_parse("${exec setsmtp -b}")
todo='work'
elseif ip == 'IP AT WORK PLACE TO' then
todo='work'
else
conky_parse("${exec setsmtp -t}")
todo='personal'
end
settings = {--IP ADDRESS
txt=ip,
x=xp+10 , y=83,
txt_weight=0 , txt_size=10 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
elseif iface == 'wlan0' then
ssid = conky_parse("${wireless_essid wlan0}")
if ssid == 'SSID WORK PLACE ONE' then
conky_parse("${exec setsmtp -b}")
todo='work'
elseif ssid == 'SSID HOME' then
conky_parse("${exec shares -a}")
conky_parse("${exec setsmtp -t}")
todo='personal'
else
conky_parse("${exec setsmtp -t}")
todo='personal'
end
settings = {--WIRELESS INFO
txt=conky_parse("${wireless_link_qual wlan0} %"),
x=xp+10 , y=83,
txt_weight=1 , txt_size=10 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
else
iface=''
end
</pre>
<p>To monitor my battery state I added this graph:</p>
<pre class="literal-block">
settings = {--BATTERY GRAPH
value=tonumber(conky_parse("${battery_percent}")),
value_max=100 ,
x=xp , y=yp ,
graph_radius=22 ,
graph_thickness=5 ,
graph_start_angle=180 ,
graph_unit_angle=2.7 , graph_unit_thickness=2.7 ,
graph_bg_colour=bgc , graph_bg_alpha=bga ,
graph_fg_colour=theme , graph_fg_alpha=fga ,
hand_fg_colour=theme , hand_fg_alpha=0.0 ,
txt_radius=0 ,
txt_weight=1 , txt_size=8.0 ,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
graduation_radius=22 ,
graduation_thickness=4 , graduation_mark_thickness=2 ,
graduation_unit_angle=27 ,
graduation_fg_colour=theme , graduation_fg_alpha=0.5 ,
caption='BATTERY' ,
caption_weight=1 , caption_size=10.0 ,
caption_fg_colour=fgc , caption_fg_alpha=fga ,
};draw_gauge_ring(settings)
settings = {--BATTERY CHARGING STATE
txt=conky_parse("${acpiacadapter} ${battery_time}"),
x=xp-25 , y=83,
txt_weight=0 , txt_size=10 ,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
</pre>
<p>As you saw in the network topic I set a todo variable based on my location. That variable will point to a specific tracks-work/tracks-personal <a class="reference external" href="https://github.com/visibilityspots/scripts#conky-trackssh">bash script</a> which will grab my todo's for work or for leisure ;)</p>
<pre class="literal-block">
-- TODO COLUMN
conky_parse("${execpi 53 ~/.conky/scripts/tracks-" .. todo .. ".sh}")
arrayYfactors={'20', '35', '51', '65'}
for i, Yfactor in ipairs(arrayYfactors) do
firstchar=conky_parse("${exec head -" .. i .. " ~/.conky/scripts/todo-" .. todo .. " | tail -1 | sed -r 's/^ //' | cut -d ' ' -f 1}")
if firstchar == '*' then
tmpweight='0'
tmpcolour=fgc
elseif firstchar == '-' then
tmpweight='0'
tmpcolour=fgc
else
tmpweight='1'
tmpcolour=theme
end
settings = { --TODO column
txt=conky_parse("${exec head -" .. i .. " ~/.conky/scripts/todo-" .. todo .. " | tail -1 | sed -r 's/^ //' | cut -d '(' -f 1}"),
x=xp+80 , y=Yfactor,
txt_weight=tmpweight , txt_size=12,
txt_fg_colour=tmpcolour , txt_fg_alpha=fga ,
};display_text(settings)
end
</pre>
<p>And last but not least based on location I monitor also a specific <a class="reference external" href="http://jenkins-ci.org/">jenkins</a> job using the <a class="reference external" href="https://github.com/Ronnie76er/conkyhudson">conky hudson</a> script</p>
<pre class="literal-block">
settings = { --JENKINS TITLE
txt=conky_parse("${exec ~/.conky/scripts/hudson/conkyhudson.py -t ~/.conky/scripts/hudson/" .. todo .. ".template | cut -d '|' -f 1 | head -1}"),
x=xp+80 , y=80,
txt_weight=1 , txt_size=9,
txt_fg_colour=theme , txt_fg_alpha=fga ,
};display_text(settings)
settings = { --JENKINS line
txt=conky_parse("${exec ~/.conky/scripts/hudson/conkyhudson.py -t ~/.conky/scripts/hudson/" .. todo .. ".template | cut -d '|' -f 2 | sed 's/_/ /' | head -1}"),
x=xp+178 , y=80,
txt_weight=0 , txt_size=10,
txt_fg_colour=fgc , txt_fg_alpha=fga ,
};display_text(settings)
</pre>
<p>The conky hudson template used in this last feature is looking like:</p>
<pre class="literal-block">
[job;1;jenkinsurl.com;nameofyourjenkinsjob]
nameofyourjenkinsjob|#[1;number] [1;result] [1;id]
</pre>
Tracks2013-05-16T19:00:00+02:002013-05-16T00:00:00+02:00Jantag:visibilityspots.org,2013-05-16:/tracks.html<p>To get an overview of my todo's I used to list them in google tasks. Back in time I was convinced it would nicely integrate with all tools software and distributions I would use. After some month's I figured out it wouldn't.</p>
<p>So I searched on the web for software which would take that task over from google. I used to play with several tools, from <a class="reference external" href="http://trac.edgewall.org/">trac</a>, to <a class="reference external" href="https://www.chiliproject.org/">chiliproject</a> to <a class="reference external" href="http://www.redmine.org/">redmine</a>.
All those tools worked very nice but were some overkill to only manages todo lists.</p>
<p>In the meantime I installed <a class="reference external" href="http://gitlab.org/">gitlabhq</a>, tried to abuse the issues there to manage …</p><p>To get an overview of my todo's I used to list them in google tasks. Back in time I was convinced it would nicely integrate with all tools software and distributions I would use. After some month's I figured out it wouldn't.</p>
<p>So I searched on the web for software which would take that task over from google. I used to play with several tools, from <a class="reference external" href="http://trac.edgewall.org/">trac</a>, to <a class="reference external" href="https://www.chiliproject.org/">chiliproject</a> to <a class="reference external" href="http://www.redmine.org/">redmine</a>.
All those tools worked very nice but were some overkill to only manages todo lists.</p>
<p>In the meantime I installed <a class="reference external" href="http://gitlab.org/">gitlabhq</a>, tried to abuse the issues there to manage my todo's. But that went into chaos when managing repo's for household tasks etc. When creating that repo I figured out it wasn't logical neither.</p>
<p>So I tumbled into a rails application. <a class="reference external" href="http://getontracks.org">Tracks</a>, and hell I like it a lot. It's easy, it can be viewed with <a class="reference external" href="http://xvx.ca/code/tracks-android/">android</a> devices, it mails every week an overview of upcoming tasks for that week and I stripped it out in my <a class="reference external" href="http://conky.sourceforge.net/">conky</a> setup.</p>
<p>Today I have 2 instances running one on my personal server and one for my tasks at the customer. Depending on location conky shows me the right tasks.</p>
<p>The setup is rather easy, it's all <a class="reference external" href="https://github.com/TracksApp/tracks/blob/v2.2.2/doc/installation.textile">explained</a> clear on their website. I opted for a sqlite3 database and running <a class="reference external" href="http://code.macournoyer.com/thin/">thin</a> to host it.</p>
<p>Nevertheless I still suffer 2 major issues, the <a class="reference external" href="https://github.com/adamwg/tracks-android/issues/20">first one</a> is related to the android app. Seems like it shows also closed tasks as being open.</p>
<p>The 2nd one is the frustration of integration github issues. Until today I didn't find a tool which is able to synchronizes all your github issues into whatever application. The only tool I found was <a class="reference external" href="https://github.com/stephencelis/ghi">ghi</a> which is just a command line overview of your github tasks.</p>
<p>So please if you found a solution for that don't hesitate to contact me about it! You could make my day!</p>
Irssi bitlbee channel2013-05-10T13:30:00+02:002013-05-10T00:00:00+02:00Jantag:visibilityspots.org,2013-05-10:/irssi-bitlbee-channel.html<p>Every time I want to join a channel on a jabber account using bitlbee I'm a bit confused and have to search the whole inter-webs before actually finding out howto configure my chat setup to do so.</p>
<p>My online search points me out to the <a class="reference external" href="http://wiki.bitlbee.org/JabberGroupchats">bitlbee wiki</a>. Nevertheless those commands never got to the point to have it actually working. After many attempts a colleague pointed me to the right solution.</p>
<p>To never forget it anymore and sharing the working setup with the world I summarize it in this blog post.</p>
<p>In your bitlbee control channel <strong>&bitlbee</strong> :</p>
<pre class="literal-block">
chat add [account …</pre><p>Every time I want to join a channel on a jabber account using bitlbee I'm a bit confused and have to search the whole inter-webs before actually finding out howto configure my chat setup to do so.</p>
<p>My online search points me out to the <a class="reference external" href="http://wiki.bitlbee.org/JabberGroupchats">bitlbee wiki</a>. Nevertheless those commands never got to the point to have it actually working. After many attempts a colleague pointed me to the right solution.</p>
<p>To never forget it anymore and sharing the working setup with the world I summarize it in this blog post.</p>
<p>In your bitlbee control channel <strong>&bitlbee</strong> :</p>
<pre class="literal-block">
chat add [account id] room@conference.jabber.link #room
/j #room
</pre>
<p>As you can see in the chat add command the ending #room was missing in the online documentation.</p>
<p>Hope to tackle many frustrations by this one :)</p>
GTalkSMS mobile alerting through xmpp protocol2012-12-19T18:56:00+01:002012-12-19T00:00:00+01:00Jantag:visibilityspots.org,2012-12-19:/gtalksms.html<p>Recently I bumped into <a class="reference external" href="http://code.google.com/p/gtalksms/">GTalkSMS</a> when I was surfing the net for manuals on irssi & bitlbee using to chat so I could move away from pidgin.</p>
<p>This GTalkSMS tool is quite cool. When am at work or at home my mobile isn't always in my sight. Therefore it could happen someone has to call me a few times before I answer the call.</p>
<p>(biggest frustration of the girlfriend meaning you have to answer your call within a time period of 3 seconds because you're working in IT)</p>
<p>Using this nifty tool you can control your mobile through your favorite chat …</p><p>Recently I bumped into <a class="reference external" href="http://code.google.com/p/gtalksms/">GTalkSMS</a> when I was surfing the net for manuals on irssi & bitlbee using to chat so I could move away from pidgin.</p>
<p>This GTalkSMS tool is quite cool. When am at work or at home my mobile isn't always in my sight. Therefore it could happen someone has to call me a few times before I answer the call.</p>
<p>(biggest frustration of the girlfriend meaning you have to answer your call within a time period of 3 seconds because you're working in IT)</p>
<p>Using this nifty tool you can control your mobile through your favorite chat client by the xmpp protocol. You install the app from the <a class="reference external" href="https://play.google.com/store/apps/details?id=com.googlecode.gtalksms">google play market</a> on your phone, choose if you want to use a <a class="reference external" href="https://code.google.com/p/gtalksms/wiki/HowToSetUp#Using_an_account_for_connecting_and_another_one_for_receiving_no">separate</a> account (preferred by the guys who developed it) I created one on jabber.org.</p>
<p>Fill those credentials into the mobile app and connect your chat client to the chosen account.</p>
<p>Once configured you can start sending commands to your phone for example the command 'where' and you get the google maps location link.</p>
<p>Have fun controlling your phone the geek way!</p>
Lighttpd change tcp port in CentOS2012-12-05T21:24:00+01:002012-12-05T00:00:00+01:00Jantag:visibilityspots.org,2012-12-05:/selinux.html<p>It seems like a very simple job, and in fact it is. But I had an issue when I tried to change this in my Cent OS 6.3 setup.</p>
<p>After some digging on the internet I found out selinux was the blocking factor.</p>
<p>The configuration of the new port has to be done in the lighttpd conf file.</p>
<p>/etc/lighttpd/lighttpd.conf</p>
<pre class="literal-block">
server.port = 2080
</pre>
<p>When I changed the config file and restarted the /etc/init.d/lighttpd service I got following error:</p>
<pre class="literal-block">
(network.c.379) can't bind to port: 2080 Permission denied
</pre>
<p>I checked that I added the …</p><p>It seems like a very simple job, and in fact it is. But I had an issue when I tried to change this in my Cent OS 6.3 setup.</p>
<p>After some digging on the internet I found out selinux was the blocking factor.</p>
<p>The configuration of the new port has to be done in the lighttpd conf file.</p>
<p>/etc/lighttpd/lighttpd.conf</p>
<pre class="literal-block">
server.port = 2080
</pre>
<p>When I changed the config file and restarted the /etc/init.d/lighttpd service I got following error:</p>
<pre class="literal-block">
(network.c.379) can't bind to port: 2080 Permission denied
</pre>
<p>I checked that I added the port to iptables, tried other ports, nothing worked. Until I found out it was related to the default selinux configuration.</p>
<p>On many forums was indicated that the problem is solved by disabling the selinux service. Nevertheless I wanted to do it the right way and after some try and error found out that by installing the package policycoreutils-python you can look up the status of the selinux feature</p>
<pre class="literal-block">
# sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: enforcing
Mode from config file: enforcing
Policy version: 24
Policy from config file: targeted
# yum provides /usr/sbin/semanage
Loaded plugins: fastestmirror, presto, priorities
Loading mirror speeds from cached hostfile
* base: centos.weepeetelecom.be
* epel: be.mirror.eurid.eu
* extras: centos.weepeetelecom.be
* remi: remi-mirror.dedipower.com
* updates: centos.weepeetelecom.be
187 packages excluded due to repository priority protections
policycoreutils-python-2.0.83-19.24.el6.x86\_64 : SELinux policy core python utilities
Repo : base
Matched from:
Filename : /usr/sbin/semanage
# yum install policycoreutils-python
</pre>
<p>Once this is done we can use the semanage command to add our new port to the selinux security feature. First we can list all already configured ports for the http service:</p>
<pre class="literal-block">
semanage port -l \| grep http\_port\_t
</pre>
<p>If the desired port isn't listed we can add this with following command:</p>
<pre class="literal-block">
semanage port -a -t http\_port\_t -p tcp 2080
</pre>
<p>You don't need to restart the selinux feature, the setting will take effect immediately after you added the last command. Once that is done you can restart the lighttpd service without the permission denied issue!</p>
<p>If you also want to have tcp port 9000 be able to work for php you also have to add this one to selinux:</p>
<pre class="literal-block">
semanage port -a -t http\_port\_t -p tcp 9000
</pre>
<p>Sources:</p>
<ul class="simple">
<li><a class="reference external" href="http://www.howtoforge.com/installing-lighttpd-with-php5-php-fpm-and-mysql-support-on-centos-6.3">http://www.howtoforge.com/installing-lighttpd-with-php5-php-fpm-and-mysql-support-on-centos-6.3</a></li>
<li><a class="reference external" href="http://www.cyberciti.biz/faq/rhel-fedora-redhat-selinux-protection/">http://www.cyberciti.biz/faq/rhel-fedora-redhat-selinux-protection/</a></li>
<li><a class="reference external" href="http://www.cyberciti.biz/faq/redhat-install-semanage-selinux-command-rpm/">http://www.cyberciti.biz/faq/redhat-install-semanage-selinux-command-rpm/</a></li>
</ul>
Dropbox2012-10-15T18:00:00+02:002012-10-15T00:00:00+02:00Jantag:visibilityspots.org,2012-10-15:/dropbox.html<p>Reading this article will go through the process I went through configuring multiple dropbox accounts on my centos machine (one personal and one for work) and encrypting them both using encfs.</p>
<p>That way I'm sure dropbox can't read the data stored into it. Because no I don't trust anybody on the cloud!</p>
<p>In the first part I will configure 2 dropbox services using a CentOS 6 Desktop, in the second part I will encrypt those 2 dropbox services using encfs.</p>
<p>The first account you can just install and configure the normal way provided by <a class="reference external" href="https://www.dropbox.com/install">dropbox</a> itself. Here I configured my …</p><p>Reading this article will go through the process I went through configuring multiple dropbox accounts on my centos machine (one personal and one for work) and encrypting them both using encfs.</p>
<p>That way I'm sure dropbox can't read the data stored into it. Because no I don't trust anybody on the cloud!</p>
<p>In the first part I will configure 2 dropbox services using a CentOS 6 Desktop, in the second part I will encrypt those 2 dropbox services using encfs.</p>
<p>The first account you can just install and configure the normal way provided by <a class="reference external" href="https://www.dropbox.com/install">dropbox</a> itself. Here I configured my work account on so all work related data will be stored in my home/Dropbox folder later on.</p>
<p>Once this is done we have to install the 2nd service and configure the personal account into it. To do this we have to create 2 new folders into your home directory.
One named .dropbox-personal where the 2nd dropbox service configuration files will be stored in. And another named for example Personal where the actual Dropbox folder for this 2nd account will be written to.</p>
<p>When you created those 2 fresh directories you can open a terminal and run the command</p>
<pre class="literal-block">
HOME=$HOME/.dropbox-personal /usr/bin/dropbox start -i
</pre>
<p>so the 2nd dropbox service will be started from this newly .dropbox-personal folder.</p>
<p>Follow the wizard, but make sure you enter a CUSTOM folder (e.g. ~/Personal) where you point to the 2nd new folder you've created ~/Dropbox-personal/!</p>
<p>To start this 2nd instance of dropbox we have to create a script which will start this instance and add it to our Startup programs.</p>
<p>Create an executable (chmod +x file.sh) .sh file with this content:</p>
<pre class="literal-block">
#!/bin/bash HOME=$HOME/.dropbox-personal /usr/bin/dropbox start
</pre>
<p>And add a new entry via System -> Preferences -> Startup Applications for this script.</p>
<p>Or you can copy the script.sh file to your /usr/bin/ director without the extension so your script will be available as a command in your terminal.</p>
<p>So right now we have 2 dropbox accounts uploading the folders ~/Dropbox for work related stuff and ~/Personal/Dropbox/ for personal data.</p>
<p>Next step will be encryption for those folders. By encrypting your dropbox folders we make sure our data will not be readable when stored on the dropbox servers and sent over the internet.
To accomplish this encryption I opted for encfs. On a CentOS machine you can install it using yum:</p>
<pre class="literal-block">
sudo yum install fuse-encfs
</pre>
<p>Once the package is installed we can configure our encrypted volumes.
The way this encryption will work is quite simple. You have 2 folders, a private folder which is the working directory where you can edit delete and create files and folders to work with.
All content in your private folder will be encrypted to you encrypted folder which will be synchronized to the dropbox online services.</p>
<p>So in our case we have to create 4 folders, 2 for our work account (~/Private & ~/Dropbox/.encrypted) and 2 for our personal account (~/Personal/Private-personal & ~/Personal/Dropbox/.encrypted-personal).</p>
<p>Once those folders are created we can configure the encfs volumes:</p>
<pre class="literal-block">
encfs ~/Dropbox/.encrypted ~/Private encfs ~/Personal/Dropbox/.encrypted-personal ~/Personal/Private-personal
</pre>
<p>where you can use the paranoia mode to encrypt your files</p>
<pre class="literal-block">
enter "p" for pre-configured paranoia mode
</pre>
<p>with your chosen password. When you created your encrypted volumes you can administer your data in the private folders, this data will be encrypted automatically to your .encrypted* folders which will be uploaded to dropbox.
When you install dropbox on another computer you also have to install encfs on it to decrypt your files.</p>
<p>In the dropbox folders you can see there is a .encfs6.xml file. To be completely sure dropbox can't do anything with your files you can exclude this file to be synchronized online.
But make some copies of it to a secure place (usb stick or your phone) before you continue. This can be done with the command:</p>
<pre class="literal-block">
dropbox exclude add ~/Dropbox/.encrypted/.encfs6.xml dropbox exclude list
</pre>
<p>And remove the .encfs6.xml file from the online dropbox account using the web service.</p>
<p>For the 2nd service you have to use the following commands:</p>
<pre class="literal-block">
HOME=$HOME/.dropbox-personal /usr/bin/dropbox exclude add ~/Personal/Dropbox/.encrypted-personal/.encfs6.xml HOME=$HOME/.dropbox-personal /usr/bin/dropbox exclude list``
</pre>
<p>On every computer where you install encfs to decrypt those files you have to copy the proper .encfs6.xml file in the .encrypted* folders so you can decrypt the encfs volumes.</p>
<p>Be aware you can't use your encrypted files using the dropbox web interface. On your android phone you can install <a class="reference external" href="https://code.google.com/p/cryptonite/">cryptonite</a> which will decrypt your files so you can use them on your phone.</p>
<p>I created a Startup script which can decrypt and umount the encrypted folders and shared it on <a class="reference external" href="https://github.com/visibilityspots/scripts">github</a> by adding the script with the preferred parameters to your Startup
programs you have to fill in the passwords each time you log in so your folders are decrypted and you can start using them.
(or add the script to your /usr/bin/ folder named encryption so you can handle it as a command called encryption as you named it in your terminal)</p>
<p>You can also use this setup to share for example your evolution mails via dropbox on an encrypted way so nobody can read your mails except your on your different computers with evolution. (Make sure the
evolution versions match and point your evolution working directory to the private one using symlinks - e.g. ln -s ~/Private/.evolution ~/.evolution)</p>
<p>Feel free to add patches, send remarks about this topic.) by adding the script with the preferred parameters to your Startup programs you have to fill in the passwords each time you log in so your folders are
decrypted and you can start using them.</p>
<p>Resources:</p>
<ul class="simple">
<li><a class="reference external" href="http://maketecheasier.com/run-multiple-dropbox-accounts-in-mac-and-linux/2010/05/24">http://maketecheasier.com</a></li>
<li><a class="reference external" href="https://help.ubuntu.com/community/EncryptedPrivateDirectory">https://help.ubuntu.com</a></li>
<li><a class="reference external" href="http://janaksingh.com/blog/dropbox-encryption-install-encfs-linux-encrypt-decrypt-dropbox-content-realtime-155">http://janaksingh.com</a></li>
</ul>
Puppet sslv3 alert certificate revoked2012-10-08T11:22:00+02:002015-10-08T00:00:00+02:00Jantag:visibilityspots.org,2012-10-08:/puppet-revoked-certificate.rst.html<p>I started the day with ssl issues using puppet. Last week I cleaned 2 hosts in our tree using the puppet command</p>
<pre class="literal-block">
# puppet node clean [hostname]
</pre>
<p>on the puppetmaster. I did this to clean out the stored configs for those nodes.</p>
<p>But I didn't realized this also cleaned out the ssl certificates for those clients. So I started the new week with this uncomfortable issue:</p>
<pre class="literal-block">
[root@agent ~]# puppet agent --test err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked warning: Not using cache on failed …</pre><p>I started the day with ssl issues using puppet. Last week I cleaned 2 hosts in our tree using the puppet command</p>
<pre class="literal-block">
# puppet node clean [hostname]
</pre>
<p>on the puppetmaster. I did this to clean out the stored configs for those nodes.</p>
<p>But I didn't realized this also cleaned out the ssl certificates for those clients. So I started the new week with this uncomfortable issue:</p>
<pre class="literal-block">
[root@agent ~]# puppet agent --test err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
</pre>
<p>After some digging on the internet I achieved to solve this issue.
Here under I described the steps to breath again:</p>
<p>To be sure the certificates are completely removed on the puppetmaster I explicitly cleaned them again</p>
<pre class="literal-block">
[root@master ~]#puppet cert -c hostname
</pre>
<p>Now we are sure those certificates are cleaned up on the master we have to do this also on the agent</p>
<p>Looking for the directory where those certificates are stored</p>
<pre class="literal-block">
[root@agent ~]# puppet --genconfig | grep certdir
# The default value is '$certdir/$certname.pem'.
# The default value is '$certdir/ca.pem'. certdir = /var/lib/puppet/ssl/certs
</pre>
<p>For older versions of puppet</p>
<pre class="literal-block">
[root@agent ~]# puppet config print | grep certdir
</pre>
<p>Removing the existing certificates on the client:</p>
<pre class="literal-block">
[root@agent ~]# rm /var/lib/puppet/ssl -rf
</pre>
<p>Once the certificates are completely removed on the master and the client we have to regenerate them from the agent using the puppet daemon</p>
<pre class="literal-block">
[root@agent ~]# puppet agent --test
</pre>
<p>or by manually regenerating them</p>
<pre class="literal-block">
[root@agent ~]# puppet certificate generate hostname.domain --ca-location remote true
</pre>
<p>As soon as new certificates are generated and we got the true back from the agent we can sign the fresh certificate on the master</p>
<p>List the certificates which are waiting to get signed and sign them</p>
<pre class="literal-block">
[root@master ~]# puppet cert -l "hostname.domain" (XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX)
[root@master ~]# puppet cert sign hostname.domain
notice: Signed certificate request for hostname.domain
notice: Removing file Puppet::SSL::CertificateRequest hostname.domain at '/var/lib/puppetmaster/ssl/ca/requests/hostname.domain.pem'
</pre>
<p>If everything went well you should be able to run puppet again on the client</p>
<pre class="literal-block">
puppet agent --test --noop
</pre>
<p>and relax again!</p>
<p>Digging the internet I crossed <a class="reference external" href="http://honglus.blogspot.be/2012/01/force-puppet-agent-to-regenerate.html">honglus blog</a> and an issue on <a class="reference external" href="http://projects.puppetlabs.com/issues/11854">puppetlabs projects</a> which made my day.</p>
Writing customized icinga checks2012-09-25T11:16:00+02:002012-09-25T00:00:00+02:00Jantag:visibilityspots.org,2012-09-25:/icinga-checks.html<p>Recently I started to try writing a customized script for the <a class="reference external" href="https://www.icinga.org/">icinga</a> monitoring tool. I will try to describe the steps I went trough to achieve this in this post. I assume you already have a working icinga setup.
If not you can find documentation about this on <a class="reference external" href="http://docs.icinga.org/">http://docs.icinga.org/</a>.</p>
<p>First of all you need to script. I created a script which will check if a service is running using the command</p>
<pre class="literal-block">
# /etc/init.d/service status
</pre>
<p>to see how to implement this in icinga. The script can be found on my <a class="reference external" href="https://github.com/visibilityspots/icinga-scripts/blob/master/check_jenkins">github</a> repo.</p>
<p>Once you have tested …</p><p>Recently I started to try writing a customized script for the <a class="reference external" href="https://www.icinga.org/">icinga</a> monitoring tool. I will try to describe the steps I went trough to achieve this in this post. I assume you already have a working icinga setup.
If not you can find documentation about this on <a class="reference external" href="http://docs.icinga.org/">http://docs.icinga.org/</a>.</p>
<p>First of all you need to script. I created a script which will check if a service is running using the command</p>
<pre class="literal-block">
# /etc/init.d/service status
</pre>
<p>to see how to implement this in icinga. The script can be found on my <a class="reference external" href="https://github.com/visibilityspots/icinga-scripts/blob/master/check_jenkins">github</a> repo.</p>
<p>Once you have tested the script you have to make sure it is copied to the scripts directory on the server you want to monitor. Usually this directory can be found in /usr/lib64/nagios/plugins/ on a CentOS 6 machine.
Also make sure your script is executable (chmod +x).</p>
<p>Next we have to configure the NRPE daemon on this remote host. The nrpe configuration file can be found in etc/nagios/nrpe.cfg found using the mlocate software.</p>
<pre class="literal-block">
# updatedb
# locate nrpe.cfg.
</pre>
<p>Here you have to make sure that you point the command to your script location by adding the underlying line and restarting the NRPE service</p>
<pre class="literal-block">
command[check_NAME]=/usr/lib64/nagios/plugins/check_NAME
/etc/init.d/nrpe restart
</pre>
<p>The server side where we have to configure the command and the service itself into the icinga service.</p>
<p>We have to add the command check_NAME into the file /etc/icinga/objects/commands.cfg</p>
<pre class="literal-block">
define command {
command_name check_NAME
command_line $USER1$/check_NAME
}
</pre>
<p>To configure the specified service you have to configure a node with this newly created command for in example /etc/icinga/objects/services/node.cfg</p>
<pre class="literal-block">
define service {
service_description DESCRIPTION
check_command check_nrpe_command!
check_NAME use generic-service
notification_period 24x7
host_name HOSTNAME.OF.SERVER
}
</pre>
<p>And restart the icinga service</p>
<pre class="literal-block">
# /etc/init.d/icinga restart
</pre>
<p>After a few minutes the check should appear into your icinga front-end. Enjoy scripting your own custom scripts!</p>
SMS server using CentOS, kannel and playsms2012-07-24T11:34:00+02:002012-07-24T00:00:00+02:00Jantag:visibilityspots.org,2012-07-24:/sms-server.html<p>On this page I will describe the way I went trough to configure an sms gateway using a laptop, <a class="reference external" href="http://www.business.vodafone.com/site/bus/public/enuk/support/10_productsupport/usb_stick/01_vodafone/02_vodafone_k3565/20_software/p_software.jsp">huawei</a> modem, <a class="reference external" href="http://www.falcom.de">falcom</a> A2D-1 or the <a class="reference external" href="http://www.option.com/support/globe-trotter-hsdpa">option</a> Globetrotter hardware using the open source software <a class="reference external" href="http://www.kannel.org/">kannel</a> & <a class="reference external" href="http://playsms.org/">playsms</a>.</p>
<p>The main goal of this project was related to the scouting movement in Belgium I'm active. We wanted to interrogate all of our members who were on a start weekend of the next scouting year. To do this we had the idea to use the sms communication channel. This because almost every youngster has the possibility to send sms messages without a big effort.</p>
<p>To …</p><p>On this page I will describe the way I went trough to configure an sms gateway using a laptop, <a class="reference external" href="http://www.business.vodafone.com/site/bus/public/enuk/support/10_productsupport/usb_stick/01_vodafone/02_vodafone_k3565/20_software/p_software.jsp">huawei</a> modem, <a class="reference external" href="http://www.falcom.de">falcom</a> A2D-1 or the <a class="reference external" href="http://www.option.com/support/globe-trotter-hsdpa">option</a> Globetrotter hardware using the open source software <a class="reference external" href="http://www.kannel.org/">kannel</a> & <a class="reference external" href="http://playsms.org/">playsms</a>.</p>
<p>The main goal of this project was related to the scouting movement in Belgium I'm active. We wanted to interrogate all of our members who were on a start weekend of the next scouting year. To do this we had the idea to use the sms communication channel. This because almost every youngster has the possibility to send sms messages without a big effort.</p>
<p>To achieve this I searched on the internet and found the <a class="reference external" href="http://playsms.org/">playsms</a> software. Using this software you can easily add an interactive flow to communicate with people using sms. We used the sms quiz where we added some questions with keywords were people could answer to and we replied with a next question.</p>
<p>As mobile provider we choose for <a class="reference external" href="http://www.mobilevikings.com">Mobile Vikings</a> a belgium operator with an open-mind. They were very helpfull when I contacted them to see if they could monitor some mobile traffic for my sim.</p>
<p>But before this software can handle your sms messages they have to be captured and received using a SIM card and pushed to the software. This step in the whole process can be done by <a class="reference external" href="http://www.kannel.org/">kannel</a>.</p>
<div class="section" id="process-using-centos-6-3-minimal-installation">
<h2>Process using CentOS 6.3 minimal installation</h2>
<p>Install some required dependency packages:</p>
<pre class="literal-block">
# yum install gcc libxml2-devel mysql-server wvdial vim
</pre>
<p>Looking for the modem:</p>
<pre class="literal-block">
# wvdialconf /etc/wvdial.conf
Found a modem on **/dev/ttyUSB0**
</pre>
</div>
<div class="section" id="minicom">
<h2>Minicom</h2>
<p>Before starting configuring the services which were going to communicate with the modem I wanted to make sure I could send text messages with it. To check that functionality I installed the <a class="reference external" href="http://linux.die.net/man/1/minicom">minicom</a> serial communication program:</p>
<pre class="literal-block">
# yum install minicom
# minicom
</pre>
<p>In this terminal you can control the modem using AT commands. A nice tutorial about those commands is available on <a class="reference external" href="http://qualityguru.wordpress.com/test-status-to-smsmms/">qualityguru</a>.</p>
<p>Steps for entering the PIN.</p>
<pre class="literal-block">
AT+CPIN=XXXX
OK
AT+CPIN?
+CPIN: READY
OK
</pre>
<p>Checking if the SMS center is configured:</p>
<pre class="literal-block">
AT+CSCA?
+CSCA: "+32XXXXXXXXX",145
OK
</pre>
<p>If not configured, configure it by:</p>
<pre class="literal-block">
AT+CSCA="+32XXXXXXXXX"
OK
</pre>
<p>The steps for sending a text message:</p>
<pre class="literal-block">
AT+CMGF=1
OK
AT+CMGS="+32XXXXXXXXX"
> This is the text message.
> (CTRL-Z)
+CMGS: XX
OK
</pre>
<p>If you received the message on your phone its working obviously and we can start configuring kannel.</p>
<p>If not, check the <a class="reference external" href="http://qualityguru.wordpress.com/2010/03/02/test-status-to-smsmms-trouble-shooting-sending-sms-messages-with-dedicated-gsm-modem-device/">troubleshoot</a> page of qualityguru for some common mistakes.</p>
</div>
<div class="section" id="kannel">
<h2>Kannel</h2>
<p>At the moment of writing this post the last stable version is 1.4.3. Using CentOS 6.4 you can install kannel from the epel <a class="reference external" href="http://fedoraproject.org/wiki/EPEL">repository</a>:</p>
<pre class="literal-block">
# yum install kannel
</pre>
<p>Or you can choose to compile it from source:</p>
<pre class="literal-block">
# wget http://www.kannel.org/download/1.4.3/gateway-1.4.3.tar.gz
# tar zxvf gateway-1.4.3.tar.gz -C /usr/local/src/
# cd /usr/local/src/gateway-1.4.3/
# mkdir -p /etc/kannel
# ./configure --prefix=/etc/kannel
# make
# make install
</pre>
<p>I installed the kannel service from the repository and created a symlink from /etc/kannel.conf to the /etc/kannel/kannel.conf so the playsms service could read the configuration afterwards:</p>
<pre class="literal-block">
# mkdir /etc/kannel/
# ln -s /etc/kannel.conf /etc/kannel/kannel.conf
</pre>
<p>Once you configured your device you start kannel by starting the kannel service:</p>
<pre class="literal-block">
# service kannel start
</pre>
<p>If everything went well you can see that there are 2 services started:</p>
<pre class="literal-block">
# ps aux | grep kannel
kannel 9611 1.9 0.1 750424 6684 ? Sl 13:14 2:37 /usr/sbin/bearerbox /etc/kannel.conf
kannel 9636 0.0 0.1 674228 4676 ? Sl 13:14 0:00 /usr/sbin/smsbox /etc/kannel.conf
</pre>
<p>In the /var/log/kannel/kannel.log file you can follow the state of the kannel service. I struggled a bit with this to find out the reset string for the modems I used. By searching the internet you can find the particular string for your device.</p>
<p>For example the option one I found on <a class="reference external" href="http://forums.enterprisemobiletoday.com/showthread.php?50854-Getting-Vodafone-s-Option-Globetrotter-to-work">enterprisemobiletoday.com</a> by try & error in the minicom terminal.</p>
<p>I used different sorts of hardware and listed the specific kannel.conf files here under per device.</p>
<p>In the first phase I used a <a class="reference external" href="http://www.business.vodafone.com/site/bus/public/enuk/support/10_productsupport/usb_stick/01_vodafone/02_vodafone_k3565/20_software/p_software.jsp">huawei</a> USB dongle:</p>
<pre class="literal-block">
#CORE
group = core
admin-port = 13000
admin-password = #PASSWORD
status-password = #PASSWORD
log-file = "/var/log/kannel/kannel.log"
log-level = 0
access-log = "/var/log/kannel/access.log"
smsbox-port = 13001
store-type = file
store-location = "/var/log/kannel/kannel.store"*
#SMSC MODEM GSM
group = smsc
smsc = at
connect-allow-ip = 127.0.0.1
port = 13013
host = "localhost"
smsc-id = Huawei
modemtype = Huawei
device = /dev/ttyUSB0
speed = 9600
sms-center = "+32486000005"
my-number = "+324XXXXXXXX"
pin = XXXX
group = modems
id = huawei
name = huawei
detect-string = "huawei"
init-string = "AT+CNMI=2,1,0,0,0;+CMEE=1"
#SMSBOX SETUP
group = smsbox
bearerbox-host = 127.0.0.1
bearerbox-port = 130X01
sendsms-port = 13131
sendsms-chars = "0123456789+"
global-sender = 00324XXXXXXXX
log-file = "/var/log/kannel/smsbox.log"
log-level = 0
access-log = "/var/log/kannel/access.log"
#SEND-SMS USERS
group = sendsms-user
username = #USERNAME
password = #PASSWORD
user-allow-ip = "\*.\*.\*.\*"
#SMS SERVICE
group = sms-service
keyword = default
accept-x-kannel-headers = true
#accepted-smsc = Huawei
accepted-smsc = at2
max-messages = 0
assume-plain-text = true
catch-all = true
get-url = "http://localhost/playsms/index.php?app=call&cat=gateway&plugin=kannel&access=geturl&t=%t&q=%q&a=%a"
</pre>
<p>During the event was in the possession of a <a class="reference external" href="http://www.falcom.de">falcom</a> A2D-1 gateway which was connected from serial to usb:</p>
<pre class="literal-block">
group = core
admin-port = 13000
admin-password = playsms
log-file = "/var/log/kannel/kannel.log"
log-level = 0
access-log = "/var/log/kannel/access.log"
smsbox-port = 13001
store-type = file
store-location = "/var/log/kannel/kannel.store"*
group = smsc
smsc = at
modemtype = falcom
device = /dev/ttyUSB0
my-number = "+324XXXXXXXX"
sms-center = "+32486000005"
pin = XXXX
group = modems
id = falcom
name = "Falcom"
detect-string = "Falcom"
reset-string = "AT+CFUN=1"
#SMSBOX SETUP
group = smsbox
bearerbox-host = localhost
sendsms-port = 13131
log-file = "/var/log/kannel/smsbox.log"
log-level = 0
access-log = "/var/log/kannel/access.log"
#SEND-SMS USERS
group = sendsms-user
username = #USER
password = #PASSWORD
#SMS SERVICE
group = sms-service
keyword = default
accept-x-kannel-headers = true
max-messages = 0
assume-plain-text = true
catch-all = true
get-url = "http://127.0.0.1:2080/playsms/index.php?app=call&cat=gateway&plugin=kannel&access=geturl&t=%t&q=%q&a=%a"
</pre>
<p>After the event I had to give back the falcom and got my hands on an <a class="reference external" href="http://www.option.com/support/globe-trotter-hsdpa">option</a> globetrotter HSPDA card connected on a pcmci slot of the laptop I configured as CentOS server:</p>
<pre class="literal-block">
#CORE
group = core
admin-port = 13000
admin-password = playsms
status-password = playsms
log-file = /var/log/kannel/kannel.log
log-level = 0
access-log = /var/log/kannel/access.log
smsbox-port = 13001
store-type = file
store-location = /var/log/kannel/kannel.store
#SMSC MODEM GSM
group = smsc
smsc = at
connect-allow-ip = 127.0.0.1
port = 13013
host = “localhost”
smsc-id = Option
modemtype = Option
device = /dev/ttyUSB0
speed = 9600
sms-center = "32486000005"
my-number = "324XXXXXXXX"
pin = XXXX
# If modemtype=auto, try everyone and defaults to this one
group = modems
id = generic
name = "Generic Modem"
reset-string = "AT&F"
#SMSBOX SETUP
group = smsbox
bearerbox-host = 127.0.0.1
bearerbox-port = 13001
sendsms-port = 13131
sendsms-chars = “0123456789+”
global-sender = 0032485550261
log-file = “/var/log/kannel/smsbox.log”
log-level = 0
access-log = “/var/log/kannel/access.log”
#SEND-SMS USERS
group = sendsms-user
username = playsms
password = playsms
#SMS SERVICE
group = sms-service
keyword = default
accept-x-kannel-headers = true
accepted-smsc = at
max-messages = 0
assume-plain-text = true
catch-all = true
</pre>
</div>
<div class="section" id="web-service">
<h2>Web service</h2>
<p>For the playsms service we need to have a webserver configured. You can use every webserver you want, I tried with xampp and lighttpd.</p>
<p>During the event I used with the xampp web service because it was working after all by following the howto of <a class="reference external" href="http://kasrut.blogspot.be/2011/07/install-playsms-and-kannel-on-centos-6.html">kasrut</a>.</p>
<p>After the event was finished I migrated to lighttpd mainly because I already had some other applications running on that service.</p>
<p><strong>Xampp</strong></p>
<pre class="literal-block">
# wget http://nchc.dl.sourceforge.net/project/xampp/XAMPP%20Linux/1.7.4/xampp-linux-1.7.4.tar.gz
# tar zxvf xampp-linux-1.7.4.tar.gz -C /opt/
# cd /opt/lampp
# ./lampp start
</pre>
<p><strong>Lighttpd</strong></p>
<p>For the installation of lighttpd I refer to a clear tutorial on <a class="reference external" href="http://www.howtoforge.com/installing-lighttpd-with-php5-php-fpm-and-mysql-support-on-centos-6.3">howtoforge</a></p>
</div>
<div class="section" id="playsms">
<h2>playsms</h2>
<p><a class="reference external" href="http://playsms.org/">playsms</a> is a free and open-source gateway. I used this software to configure a big quiz to set up a cool and trendy communication flow between people and our scouting movement.</p>
<p>I used the <a class="reference external" href="https://github.com/antonraharja/playSMS">git</a> repository to easily update my instance to the newest releases:</p>
<pre class="literal-block">
# cd /usr/local/src/
# git clone git@github.com:antonraharja/playSMS.git
# cd playSMS/
</pre>
<p>Creation of the necessary directories and copy the web files to the webserver directory</p>
<pre class="literal-block">
# mkdir -p /var/www/html/playsms /var/spool/playsms /var/log/playsms /var/lib/playsms
# cp -r usr/local/src/playSMS/web/* /var/www/html/playsms/
</pre>
<p>Creation of a mysql db and user:</p>
<pre class="literal-block">
# mysql -u root -p
# Enter password:
# mysql> create database playsms;
Query OK, 1 row affected (0.00 sec)
# mysql> grant usage on *.* to USER@localhost identified by ‘PASSWORD’;
Query OK, 0 rows affected (0.00 sec)
# mysql> grant all privileges on playsms.* to USER@localhost ;
Query OK, 0 rows affected (0.00 sec)
# mysql> quit
# msql -u root -p playsms < /usr/local/src/playSMS/db/playsms.sql
</pre>
<p>Next step is to configure the playsms web service. Therefore follow those steps:</p>
<pre class="literal-block">
# cd /var/www/html/playsms
# cp config-dist.php config.php
</pre>
<p>Edit this config.php file to your own needs.</p>
<p>Now we configured the parameters we can start to install the services:</p>
<pre class="literal-block">
# mkdir -p /etc/default /usr/local/bin
# cp /usr/local/src/playSMS/daemon/linux/etc/playsms /etc/default/
# cp /usr/local/src/playSMS/daemon/linux/bin/* /usr/local/bin/
# vim /etc/default/playsms # edit the paths to your environment ones
</pre>
<p>I've used rc.local to start the service at boot:</p>
<pre class="literal-block">
# vim /etc/rc.d/rc.local
</pre>
<p>and put /usr/local/bin/playsmsd_start at the end of that file</p>
<p>Next I configured 2 new aliases in my ~/.bashrc to easily start and stop the service:</p>
<pre class="literal-block">
alias playsms-start='/usr/local/bin/playsmsd_start'
alias playsms-stop='/usr/local/bin/playsmsd_stop'
</pre>
<p>By re-logging in you can start the service by:</p>
<pre class="literal-block">
# playsms-start
</pre>
<p>And check if the necessary services are started:</p>
<pre class="literal-block">
# ps aux | grep playsms
root 7735 0.0 0.0 103236 868 pts/4 S+ 15:52 0:00 grep playsms
root 21845 0.0 0.0 106312 1660 pts/4 S 12:25 0:06 /bin/bash ./playsmsd /var/www/html/playsms
root 21847 0.0 0.0 106184 1536 pts/4 S 12:25 0:05 /bin/bash ./sendsmsd /var/www/html/playsms
</pre>
<dl class="docutils">
<dt>Finally you can browse <a class="reference external" href="http:/">http:/</a>/<your web server IP>/playsms/ and login using</dt>
<dd>username: admin
password: admin</dd>
</dl>
<p>Where you need to configure kannel in the menu: Gateway > Manage Kannel > kannel (Inactive) (click here to activate) and adopt the parameters to the ones you configured in kannel.conf</p>
<p>After filling in your preferences you should be able to send and receive messages through this nifty web console.</p>
<p>(TIP: Using twice the same keyword for a quiz resulted in the fact that only this word is needed to send to the sms server to start the interactivity)</p>
<p>Have fun with it!</p>
</div>
SSH acces on Lacie Network Space 22012-05-28T15:18:00+02:002012-05-28T00:00:00+02:00Jantag:visibilityspots.org,2012-05-28:/lacie-networkspace-2.html<p>Recently we installed a Lacie Network Space 2 at home. Easy to share documents on the LAN network, having a central place for common media etc. After playing around with it I wanted to see if it's possible to gain access to the underlying operating system of it. On that way I could for example use this access to wake up a pc with wake on LAN.</p>
<p>And guess what, it can be done and thanks to a script of a guy Andreus it's even very easy! I found a <a class="reference external" href="http://forum.nas-central.org/viewtopic.php?f=240&t=4631">forum post</a> about his work and tested it successfully with …</p><p>Recently we installed a Lacie Network Space 2 at home. Easy to share documents on the LAN network, having a central place for common media etc. After playing around with it I wanted to see if it's possible to gain access to the underlying operating system of it. On that way I could for example use this access to wake up a pc with wake on LAN.</p>
<p>And guess what, it can be done and thanks to a script of a guy Andreus it's even very easy! I found a <a class="reference external" href="http://forum.nas-central.org/viewtopic.php?f=240&t=4631">forum post</a> about his work and tested it successfully with the latest firmware version 2.2.8!</p>
<p>After you created the right capsule the best way do update is to force it <a class="reference external" href="http://lacie.nas-central.org/wiki/Category:2big_Network_2#3._Manual_Force_Update">manually</a>.</p>
<p>When you achieved to get ssh access to the device you can play around with it, by for example installing debian squeeze in a chkroot environment. On the <a class="reference external" href="http://lacie.nas-central.org/wiki/Category:Network_Space_2">wiki</a> of nas-central.org you can find more information how to play around with your ssh access.</p>
<p>Have fun with it!</p>
Create and distribute .rpm package2012-04-07T17:31:00+02:002012-04-07T00:00:00+02:00Jantag:visibilityspots.org,2012-04-07:/rpm-package.html<p>You wrote a piece of software and want to distribute it on an easy way through a yum repository? That can be done, by making in the first place an rpm package of your code.</p>
<p>In the first place you need to set up a directory structure. This can be done using the tool rpmdevtools on a rhel based machine:</p>
<pre class="literal-block">
# yum install rpmdevtools
</pre>
<p>Once you installed the software you need to setup the directory tree:</p>
<pre class="literal-block">
$ rpmdev-setuptree
</pre>
<p>This will install the necessary rmpbuild directory tree.</p>
<p>You will see there is create a SOURCES directory, you need to get your software source …</p><p>You wrote a piece of software and want to distribute it on an easy way through a yum repository? That can be done, by making in the first place an rpm package of your code.</p>
<p>In the first place you need to set up a directory structure. This can be done using the tool rpmdevtools on a rhel based machine:</p>
<pre class="literal-block">
# yum install rpmdevtools
</pre>
<p>Once you installed the software you need to setup the directory tree:</p>
<pre class="literal-block">
$ rpmdev-setuptree
</pre>
<p>This will install the necessary rmpbuild directory tree.</p>
<p>You will see there is create a SOURCES directory, you need to get your software source into here. Best is to copy your source code into this directory named like * namoOfYourSoftware-0.0* where 0.0 stands for the release version.</p>
<p>Once you copied your source code you need to package it into a tar file:</p>
<pre class="literal-block">
$ tar -cvzf nameOfYourSoftware-0.0.tar.gz namoOfYourSoftware-0.0
</pre>
<p>Once you packaged your source code we need to create a <a class="reference external" href="http://kmymoney2.sourceforge.net/phb/rpm-example.html">spec file</a> in the SPEC directory.</p>
<p>When you created and configured your spec file the last thing we need to do is to build the actual rpm package:</p>
<pre class="literal-block">
$ rpmbuild -ba SPECS/name.spec
</pre>
<p>If everything went smooth you should find your rpm package in the RPMS directory.</p>
<p>To install your rpm package to see if it actually works:</p>
<pre class="literal-block">
rpm -ivh name-package.rpm
</pre>
<p>Now you have your own rpm package you can distribute. A nice and clean distribution solution could be your very own <a class="reference external" href="http://yum.baseurl.org/wiki/RepoCreate">yum repository</a></p>
<p>Resources:</p>
<ul class="simple">
<li><a class="reference external" href="http://rpmfind.net/linux/rpm2html/search.php?query=hello&submit=Search+...">hello world</a> package</li>
<li><a class="reference external" href="http://tldp.org/HOWTO/RPM-HOWTO/index.html">tutorial</a> from The Linux Documentation project</li>
</ul>
Puppet module mumble-server2012-04-04T15:31:00+02:002012-04-04T00:00:00+02:00Jantag:visibilityspots.org,2012-04-04:/puppet-mumble.html<p><a class="reference external" href="http://mumble.sourceforge.net/">Mumble</a> is an open source, low-latency, high quality voice chat software primarily intended for use while gaming.</p>
<p><a class="reference external" href="http://puppetlabs.com/">Puppet</a> is a tool designed to manage the configuration of Unix-like and Microsoft Windows systems decoratively.</p>
<p>The <a class="reference external" href="https://github.com/visibilityspots/puppet-mumble">puppet-mumble</a> module installs a mumble server (version 1.2.3) automatically on a CentOS 6.x machine using the puppet software based on <a class="reference external" href="http://mumble.sourceforge.net/Install_CentOS5">mumble-documentation</a>.</p>
<p>The module needs a repository which contains the <a class="reference external" href="http://www.visibilityspots.com/repos/repoview/mumble-server.html">mumble-server</a> package. I distribute this package on my own <a class="reference external" href="http://www.visibilityspots.com/repos/repoview/">visibilityspots</a> repository.</p>
<p>Using puppet this will create the necessary mumble user and group and will configure the mumble-server using your desired settings, like username, password …</p><p><a class="reference external" href="http://mumble.sourceforge.net/">Mumble</a> is an open source, low-latency, high quality voice chat software primarily intended for use while gaming.</p>
<p><a class="reference external" href="http://puppetlabs.com/">Puppet</a> is a tool designed to manage the configuration of Unix-like and Microsoft Windows systems decoratively.</p>
<p>The <a class="reference external" href="https://github.com/visibilityspots/puppet-mumble">puppet-mumble</a> module installs a mumble server (version 1.2.3) automatically on a CentOS 6.x machine using the puppet software based on <a class="reference external" href="http://mumble.sourceforge.net/Install_CentOS5">mumble-documentation</a>.</p>
<p>The module needs a repository which contains the <a class="reference external" href="http://www.visibilityspots.com/repos/repoview/mumble-server.html">mumble-server</a> package. I distribute this package on my own <a class="reference external" href="http://www.visibilityspots.com/repos/repoview/">visibilityspots</a> repository.</p>
<p>Using puppet this will create the necessary mumble user and group and will configure the mumble-server using your desired settings, like username, password, and tcp port the daemon will listen on.</p>
Cisco HWIC 3G configuration to 2G2012-04-03T14:36:00+02:002012-04-03T00:00:00+02:00Jantag:visibilityspots.org,2012-04-03:/cisco-3g.html<p>In some cisco routing devices you have the possibility to extend the features with a HWIC 3G card so mobile connectivity is added to your network infrastructure. This can be interesting for a mobile fail-over connection.</p>
<p>But as we all now, the mobile reception isn't always that good. To see the signal strength on your cisco device you can use:</p>
<pre class="literal-block">
show cellular 0/0/0 connection
</pre>
<p>depending on which slot you plugged the HWIC card into. If the measured value is beneath the -100 dBm then you have sufficient signal to setup a 3G ( CDMA - WCDMA) connection on.</p>
<p>If that's …</p><p>In some cisco routing devices you have the possibility to extend the features with a HWIC 3G card so mobile connectivity is added to your network infrastructure. This can be interesting for a mobile fail-over connection.</p>
<p>But as we all now, the mobile reception isn't always that good. To see the signal strength on your cisco device you can use:</p>
<pre class="literal-block">
show cellular 0/0/0 connection
</pre>
<p>depending on which slot you plugged the HWIC card into. If the measured value is beneath the -100 dBm then you have sufficient signal to setup a 3G ( CDMA - WCDMA) connection on.</p>
<p>If that's not the case or the values of different measurements are very different you should consider to downgrade too a 2G connection because else your 3G connection will be very wonky!</p>
<p>The default setup would mostly be auto-band. This means that if there is a little chance to connection over 3G your device will try to connect to this 3G connection. If the 3G signals gets lost it REconnects to 2G and therefore connectivity interruption will take place.</p>
<pre class="literal-block">
cellular 0/0/0 gsm band auto-band
</pre>
<p>So if your 3G signals isn't strong enough it's a good idea to force your device to connect only on 2G. This can easily be done by:</p>
<pre class="literal-block">
cellular 0/0/0 gsm band gsm-all-bands
</pre>
<p>That way your wonky 3G connection will became a stable 2G connection!</p>
BGP announcing default route2012-03-31T13:07:00+02:002012-03-31T00:00:00+02:00Jantag:visibilityspots.org,2012-03-31:/bgp-default-route.html<div class="section" id="advertising-default-route-with-bgp">
<h2>Advertising default route with BGP</h2>
<p>If you want to announce the default route which is statically routed then you have to add following commands to the working BGP configuration:</p>
<pre class="literal-block">
ip route 0.0.0.0 0.0.0.0 192.168.1.1
router bgp 65001
network 0.0.0.0
default-information originate
</pre>
<p>when you then clear the ip bgp routing softly (so the current connecting will not be broken)</p>
<pre class="literal-block">
clear ip bgp soft in
clear ip bgp soft out
</pre>
<p>you should see that the default route is will be advertised:</p>
<pre class="literal-block">
sh ip bgp summary
sh ip bgp neighbors IP …</pre></div><div class="section" id="advertising-default-route-with-bgp">
<h2>Advertising default route with BGP</h2>
<p>If you want to announce the default route which is statically routed then you have to add following commands to the working BGP configuration:</p>
<pre class="literal-block">
ip route 0.0.0.0 0.0.0.0 192.168.1.1
router bgp 65001
network 0.0.0.0
default-information originate
</pre>
<p>when you then clear the ip bgp routing softly (so the current connecting will not be broken)</p>
<pre class="literal-block">
clear ip bgp soft in
clear ip bgp soft out
</pre>
<p>you should see that the default route is will be advertised:</p>
<pre class="literal-block">
sh ip bgp summary
sh ip bgp neighbors IP.ADDRESS advertised-routes
</pre>
</div>
Apple remote (A1156) - MacBook Pro 3.1 & Ubuntu 10.042011-01-27T22:47:00+01:002011-01-27T00:00:00+01:00Jantag:visibilityspots.org,2011-01-27:/apple-linux-remote.html<p>It isn't supported by default using Ubuntu but it's as handy as hell, the apple infrared remote control. After some mayor headaches I finally succeeded to configure it manually on my MacBook Pro 3.1 running Ubuntu 10.04.</p>
<p>It's quite easy once you know how.</p>
<p>Installation of the lirc library:</p>
<pre class="literal-block">
$ sudo apt-get install lirc
</pre>
<p>Adapting the configuration files (make sure to backup them first!):</p>
<pre class="literal-block">
$ sudo cp /old/file /new/file.bak
</pre>
<p>/etc/lirc/hardware.conf</p>
<pre class="literal-block">
# /etc/lirc/hardware.conf # #Chosen Remote Control REMOTE="Apple Mac mini USB IR Receiver" REMOTE_MODULES="uinput" REMOTE_DRIVER="macmini" REMOTE_DEVICE="/dev/usb/hiddev0" REMOTE_SOCKET="" REMOTE_LIRCD_CONF …</pre><p>It isn't supported by default using Ubuntu but it's as handy as hell, the apple infrared remote control. After some mayor headaches I finally succeeded to configure it manually on my MacBook Pro 3.1 running Ubuntu 10.04.</p>
<p>It's quite easy once you know how.</p>
<p>Installation of the lirc library:</p>
<pre class="literal-block">
$ sudo apt-get install lirc
</pre>
<p>Adapting the configuration files (make sure to backup them first!):</p>
<pre class="literal-block">
$ sudo cp /old/file /new/file.bak
</pre>
<p>/etc/lirc/hardware.conf</p>
<pre class="literal-block">
# /etc/lirc/hardware.conf # #Chosen Remote Control REMOTE="Apple Mac mini USB IR Receiver" REMOTE_MODULES="uinput" REMOTE_DRIVER="macmini" REMOTE_DEVICE="/dev/usb/hiddev0" REMOTE_SOCKET="" REMOTE_LIRCD_CONF="" REMOTE_LIRCD_ARGS="--uinput"
#Chosen IR Transmitter
TRANSMITTER="None"
TRANSMITTER\_MODULES=""
TRANSMITTER\_DRIVER=""
TRANSMITTER\_DEVICE=""
TRANSMITTER\_SOCKET=""
TRANSMITTER\_LIRCD\_CONF=""
TRANSMITTER\_LIRCD\_ARGS=""
#Enable lircd
START\_LIRCD=true
#Don't start lircmd even if there seems to be a good config file
#START\_LIRCMD="false"
#Try to load appropriate kernel modules
LOAD\_MODULES="true"
# Default configuration files for your hardware if any
LIRCMD\_CONF=""
#Forcing noninteractive reconfiguration
#If lirc is to be reconfigured by an external application
#that doesn't have a debconf frontend available, the noninteractive
frontend can be invoked and set to parse REMOTE and TRANSMITTER
#It will then populate all other variables without any user input
#If you would like to configure lirc via standard methods, be sure
#to leave this set to "false"
FORCE\_NONINTERACTIVE\_RECONFIGURATION="false"
START\_LIRCMD=""
# Remote settings required by gnome-lirc-properties
REMOTE\_MODEL="A1156"
REMOTE\_VENDOR="Apple"
# Receiver settings required by gnome-lirc-properties
RECEIVER\_MODEL="Built-in\\ IR\\ Receiver\\ \\(0x8242\\)"
RECEIVER\_VENDOR="Apple"
**/etc/lirc/lircd.conf**
``# This configuration has been automatically generated # by the GNOME LIRC Properties control panel. # # Feel free to add any custom remotes to the configuration # via additional include directives or below the existing # include directives from your selected remote and/or # transmitter. #``
# Configuration selected with GNOME LIRC Properties
# include
begin remote
name AppleRemote
bits 8
eps 30
aeps 100
one 0 0
zero 0 0
pre\_data\_bits 24
pre\_data 0x87EE81
gap 211982
toggle\_bit\_mask 0x0
ignore\_mask 0x0000ff01
begin codes
KEY\_VOLUMEUP 0x0B
KEY\_VOLUMEDOWN 0x0D
KEY\_PREVIOUSSONG 0x08
KEY\_NEXTSONG 0x07
KEY\_PLAYPAUSE 0x04
KEY\_MENU 0x02
end codes
end remote
</pre>
<p>/etc/modules</p>
<pre class="literal-block">
# /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored.
lp
usbhid
applesmc
</pre>
<p>/etc/modprobe.d/blacklist</p>
<pre class="literal-block">
blacklist applesmc blacklist usbhid
</pre>
<p>Restart the lirc daemon after adopted the configuration:</p>
<pre class="literal-block">
$ /etc/init.d/lirc restart
</pre>
<p>To see if the daemon successfully started and is using the right driver:</p>
<pre class="literal-block">
$ ps aux | grep lirc
</pre>
<p>If everything went well you should be able to use the remote without any hassle and you could use the apple hardware user experience on a linux distribution!</p>
Symbolic linux links2010-02-10T00:00:00+01:002010-02-10T00:00:00+01:00Jantag:visibilityspots.org,2010-02-10:/symbolic-links.html<p>It's rather simple, but I used to look for it a while when writing my first bash/python scripts. Wanted to typing in one command so I would need to type in every time the whole path to my newly written script.</p>
<p>That way routine tasks could be called much faster and easier. This can be done by creating a symlink to your /usr/bin directory:</p>
<pre class="literal-block">
ln -s /path/to/your/script /usr/bin/nameOfTheOverallCommmandYouWantToUseForYourScript
</pre>
Permissions website2010-02-09T18:08:00+01:002010-02-09T00:00:00+01:00Jantag:visibilityspots.org,2010-02-09:/web-permissions.html<p>The most recommended permissions for files and directories on the web are 0755 and 0644. If you have shell access to your webserver you can set those permissions using those commands:</p>
<pre class="literal-block">
find -type d -print0 | xargs -0 chmod 755
find -type f -print0 | xargs -0 chmod 644
</pre>
Conky2009-12-30T14:18:00+01:002009-12-30T00:00:00+01:00Jantag:visibilityspots.org,2009-12-30:/conky.html<p>To monitor the different resources of my local system I use conky. After you installed the conky software you can start with the configuration of it.</p>
<pre class="literal-block">
$ apt-get install conky conky-colors
</pre>
<p>After I adapted the configuration my desktop became like this:</p>
<a class="reference external image-reference" href="images/conky/desktop.png"><img alt="Desktop image" src="images/conky/desktop.png" /></a>
<p>At the left side there is a pane which only monitors my system resources. The config file for it, <a class="reference external" href="http://www.visibilityspots.com/documents/conky/conkyrc">conkyrc</a> should be placed in your home directory as a hidden file (naming it .conkyrc).</p>
<p>When you now type in conky in your terminal, you should see appearing the pane on your desktop:</p>
<a class="reference external image-reference" href="images/conky/conky.png"><img alt="Left panel" src="images/conky/conky.png" /></a>
<p>On the right bottom I created an rss …</p><p>To monitor the different resources of my local system I use conky. After you installed the conky software you can start with the configuration of it.</p>
<pre class="literal-block">
$ apt-get install conky conky-colors
</pre>
<p>After I adapted the configuration my desktop became like this:</p>
<a class="reference external image-reference" href="images/conky/desktop.png"><img alt="Desktop image" src="images/conky/desktop.png" /></a>
<p>At the left side there is a pane which only monitors my system resources. The config file for it, <a class="reference external" href="http://www.visibilityspots.com/documents/conky/conkyrc">conkyrc</a> should be placed in your home directory as a hidden file (naming it .conkyrc).</p>
<p>When you now type in conky in your terminal, you should see appearing the pane on your desktop:</p>
<a class="reference external image-reference" href="images/conky/conky.png"><img alt="Left panel" src="images/conky/conky.png" /></a>
<p>On the right bottom I created an rss feeds pane. In the file <a class="reference external" href="http://www.visibilityspots.com/documents/conky/conkyrc2">conkyrc2</a> some parameters needs your attention. After you found your rss feeds, you can displays them by this instruction (10 stands for the refresh interval in minutes, 5 for the last 5 items of your feed):</p>
<pre class="literal-block">
${rss http://jouwfeed 10 item_titles 5}
</pre>
<p>Once you adopted the file and placed in as a hidden file in your home directory you can start the monitor by:</p>
<pre class="literal-block">
conky -c ~/.conkyrc2
</pre>
<p>Using the parameter -d you can force the service to start up as a background daemon process:</p>
<pre class="literal-block">
conky -c ~/.conkyrc2 -d
</pre>
<a class="reference external image-reference" href="images/conky/conkyrc2.png"><img alt="Right bottom panel" src="images/conky/conkyrc2.png" /></a>
<p>The last pane, on the right top, I configured with for monitoring my social networks and communication. In the <a class="reference external" href="http://www.visibilityspots.com/documents/conky/conkyrc3">conkyrc3</a> file you need to adopt the twitter and linkedin feed.</p>
<p>For the twitter rss feed you need 3 params: username, password, and a token. To find your token, surf to and click on the right bottom to get your rss feed. In your browser url address bar you can find the token at the end of the url XXXXXXXX.rss.</p>
<pre class="literal-block">
${rss http://twitternaam:twitterwachtwoord@twitter.com/statuses/friends_timeline/twittercode.rss 20 item_titles 2}
</pre>
<p>For the linkedin rss feed, you need to log in on linkedin.com and search for your own rss feed on the homepage. To display pidgin statuses I used the <a class="reference external" href="http://ubuntuforums.org/showthread.php?t=969933">conkyPidgin</a> module.</p>
<p>After you adopted the last configuration file once again, place it in your homedir, hide it and start it in your terminal:</p>
<pre class="literal-block">
conky -c ~/.conkyrc2
</pre>
<a class="reference external image-reference" href="images/conky/conkyrc3.png"><img alt="Right top panel" src="images/conky/conkyrc3.png" /></a>
<p>To automatically start up the conky daemons, you could call a .startConky.sh script with this code:</p>
<pre class="literal-block">
#!/bin/bash
sleep 30 && conky -d;
sleep 40 && conky -c ~/.conkyrc2 -d;
sleep 50 && conky -c ~/.conkyrc3 -d;
chmod +x .startConky.sh
</pre>
<p>Then add this command to menu System - Preferences - Startup Applications:</p>
<pre class="literal-block">
~/.start_conky.sh
</pre>
<p>This way you also could monitor your system in a fancy way :)</p>
Showing birthdays using php2009-12-30T11:27:00+01:002009-12-30T00:00:00+01:00Jantag:visibilityspots.org,2009-12-30:/birthday-script.html<p>For our local scouting group it seemed nice to write a birthday script which displays for every member on the day of his/her birthday the name and age on our homepage.</p>
<p>Something like "We wish XXX a happy # anniversary!'</p>
<p>To accomplish this I wrote a php script which gets the data of our members from a mysql db and shows the messages on the right day on our website. In the meantime also an automatic mail will be send to the person with some sort of 'personal' message.</p>
<p>database connection (db_connectPear.php)</p>
<pre class="code php literal-block">
<span class="x">< ?php
$dsn = array(
'phptype' => 'mysql',
'username' => 'DBUSERNAME' …</span></pre><p>For our local scouting group it seemed nice to write a birthday script which displays for every member on the day of his/her birthday the name and age on our homepage.</p>
<p>Something like "We wish XXX a happy # anniversary!'</p>
<p>To accomplish this I wrote a php script which gets the data of our members from a mysql db and shows the messages on the right day on our website. In the meantime also an automatic mail will be send to the person with some sort of 'personal' message.</p>
<p>database connection (db_connectPear.php)</p>
<pre class="code php literal-block">
<span class="x">< ?php
$dsn = array(
'phptype' => 'mysql',
'username' => 'DBUSERNAME',
'password' => 'DBPASSWORD',
'hostspec' => 'localhost',
'database' => $database,
);
$db_object = DB::connect($dsn, TRUE);
if(DB::isError($db_object)) {
die($db_object->getMessage());
}
$db_object->setFetchMode(DB_FETCHMODE_ASSOC);
?></span>
</pre>
<p>birthday script (birthday.php)</p>
<pre class="code php literal-block">
<span class="x">< ?php
$database = 'DBNAME';
require 'db_connectPear.php';
$selectLeden = 'SELECT *, round((to_days(now())-to_days(Geboortedatum))/365) as leeftijd
FROM `ledenlijst` WHERE '
. ' DAYOFMONTH(Geboortedatum)=date_format(now(),\'%d\') AND '
. ' MONTH(Geboortedatum)=date_format(now(),\'%c\') ORDER BY voornaam ASC';
$queryLeden = mysql_query($selectLeden)or die(mysql_error());
$selectStam = 'SELECT *, round((to_days(now())-to_days(Geboortedatum))/365) as leeftijd
FROM `stam` WHERE '
. ' DAYOFMONTH(Geboortedatum)=date_format(now(),\'%d\') AND '
. ' MONTH(Geboortedatum)=date_format(now(),\'%c\') ORDER BY voornaam ASC';
$queryStam = mysql_query($selectStam)or die(mysql_error());
if (mysql_num_rows($queryLeden) == 0) {
if (mysql_num_rows($queryStam) == 0) {
} else {
echo " Wij wensen";
while($list = mysql_fetch_object($queryStam)){
$naam = $list->voornaam." ".$list->achternaam;
echo ("$naam, ");
if ($list->mailVerjaardag == 'n') {
$tekst="Beste ".$list->voornaam."\n \n Een gelukkige verjaardag, ".$list->leeftijd ." jaar is niet niks, geniet van deze mooie dag. \n \n De leiding \n ";
$email = $list->email;
$onderwerp="Gelukkige Verjaardag!";
$headers = "From: Naam \r\n";
mail($email,$onderwerp,$tekst,$headers);
$sql = "UPDATE stam SET mailVerjaardag = 'y' WHERE id = '$list->id'";
mysql_query($sql)or die(mysql_error());
}
}
echo "een gelukkige verjaardag!";
}
} else {
echo " Wij wensen ";
$setIntro = 1;
while($list = mysql_fetch_object($queryLeden)){
$naam = $list->voornaam." ".$list->achternaam;
echo ("$naam - ($list->leeftijd jaar), ");
if ($list->mailVerjaardag == 'n') {
$tekst="Beste ".$list->voornaam."\n \n Een gelukkige verjaardag, ".$list->leeftijd ." jaar is niet niks, geniet van deze mooie dag. \n \n De leiding";
$email = $list->email;
$onderwerp="Gelukkige Verjaardag!";
$headers = "From: Naam \r\n";
mail($email,$onderwerp,$tekst,$headers);
$sql = "UPDATE ledenlijst SET mailVerjaardag = 'y' WHERE ledenlijst_id = '$list->ledenlijst_id'";
mysql_query($sql)or die(mysql_error());
}
}
if (mysql_num_rows($queryStam) == 0) {
} else {
if ($setIntro != 1){
echo " Wij wensen ";
}
while($list = mysql_fetch_object($queryStam)){
$naam = $list->voornaam." ".$list->achternaam;
echo ("$naam, ");
if ($list->mailVerjaardag == 'n') {
$tekst="Beste ".$list->voornaam."\n \n Een gelukkige verjaardag, ".$list->leeftijd ." jaar is niet niks, geniet van deze mooie dag. \n \n De leiding";
$email = $list->email;
$onderwerp="Gelukkige Verjaardag!";
$headers = "From: Naam \r\n";
mail($email,$onderwerp,$tekst,$headers);
$sql = "UPDATE stam SET mailVerjaardag = 'y' WHERE id = '$list->id'";
mysql_query($sql)or die(mysql_error());
}
}
}
echo "een gelukkige verjaardag!";
}
?></span>
</pre>