Getting top-N elements in Spark

May 11, 2019


The documentation for pyspark top() function has this warning:

This method should only be used if the resulting array is expected
to be small, as all the data is loaded into the driver's memory.

This piqued my interest: why would you need to bring all the data to the driver, if all you need is a few top elements?

The answer is: it does not load all the data into the driver’s memory.

Livy is out of memory

Mar 11, 2018

spark livy azure

Spark jobs were failing. All of them. The data pipeline had stopped. This is a tale of high-pressure debugging.

Accessing home computer from anywhere

Nov 26, 2017

mac remotedesktop ssh tunnel

Do you sometimes want to access your home computer from an outside network? Maybe you use another system, but you do not trust it and would prefer your home computer for some workflows?

This post outlines the steps to make such access possible.

The program that would not go away

Feb 17, 2017

unix signals fork exec

This post is about a program hang. The hang was in the Python process that was running Ansible scripts. The problem was hard to debug and had me go back to Unix textbook.

Correct way to create a directory in Python

Feb 2, 2017

python mkdir unix concurrency

Can you see the problem with this code? It comes from Ansible, v2.1.1.0.

if not os.path.exists(value):
     os.makedirs(value, 0o700)

It’s quite straightforward. It checks if a directory path exists. If it does not, then it creates the directory path, similar to mkdir -p. What could be wrong?

Getting rid of unused virtual disks on XenServer

Feb 1, 2017

xenserver virtualization shell-scripts

A continuous test server I’d set up had stopped working. The XenServer on which it was running had a 1TB disk: and it was full. What’s going on?

Log rotation, no code change needed

Jan 24, 2017

freebsd syslog logrotate newsyslog

This post shows you how to rotate old logs from your application. There is no change to application code. There is no specialized logging library or framework needed. It works for any language, on standard Unix platform.

Resetting a TCP connection and SO_LINGER

Oct 21, 2016

tcp freebsd networking

Can you quickly close a TCP connection in your program by sending a reset (“RST”) packet?

Program crashes, only in 64-bit mode

Oct 14, 2016

amd64 gcc c x86 asm

This is a story of a program that worked, until it broke on a 64-bit platform.

ps | grep is broken on FreeBSD

Oct 10, 2016

ps grep freebsd shell

It is true. Even on the latest FreeBSD 11.0 (I checked the source tree).

Debugging emacs slowness

Sep 29, 2016

emacs elisp debugging python

I use Emacs (emerge) while merging files. Today, when trying to merge some Python code, I found that it was taking exceedingly long time. It was blank for 5 minutes and counting.

Weird PID Files

Sep 29, 2016

unix daemons debugging shell-scripts

This is a story of multiple processes running on a system, but with empty PID files.

It took a lot of debugging.

Don't hang up on me

Jul 30, 2016

signals shell unix sighup

Can you tell me when a shell sends a “HANGUP” signal to a process? What happens if there is a pipeline? What if you prefix this pipeline with the “nohup” command?

Rotating the log makes it empty

Jul 29, 2016

logrotate syslog freebsd

If you don’t do log rotation right, you may have a full hard drive and a ghost file.

Socket timeout

Jun 29, 2016

hadoop hdfs java networking debugging

Have you run into a problem that happens only once in a while, and you never seem to have enough information to figure it out? We all have. This is one such tale: except that it has a happier ending.

Build fails, only for you

Jun 16, 2016

build autoconf gcc ld

As programmers, we face it many times: a build fails on only our system. This is one such tale.

You got too many open files

May 21, 2016

python ansible freebsd posix ipc semaphores

This is a tale about a hopelessly bad error message.