Session
FOSDEM Schedule 2021
Perl and Raku Programming

Hypers and Gathers and Takes! Oh my!

Processing large datasets in Raku.
D.perl
Steven Lembark
Raku takes the pain out of parallel processing. Combined with rational numbers for lossless conversions it is an ideal language for ETL. The operators may take a bit getting used to for Perl hackers, however. This talk looks at pre-processing the nr.gz file, splitting it into separate streams of digests and data to efficiently find duplicates. This walks through the basic process of getting the file open with gzip, acquiring data with block reads, gathers and takes, lazy operators, and other bits of technological magick. The goal is showing a reasonable framework for parallel processing of a really large dataset.

No, Raku isn't Perl. But it is high-level and offers quite a bit of 20/20 hindsight. The new constructs make it rather well suited to handling ETL, especially transforming bulky datasets. Finding duplicates in the nr.gz file is a multi-stage process, the first pre-processing stage can take hours. Going through a dozen iterations of the code I've found some alternatives for reading and parallelizing the data that could save re-inventing some wheels. Much of the linguistic territory here is foreign to Perl: gathers and takes, hypers, lazy operators, even the maps read backward (well, forward, but that's backward!).

Much of the programming involves tradeoffs between speed and size, as usual, so I'll also look at a few different ways to chunk and re-process the data for speed or space.

Additional information

Type devroom

More sessions

2/6/21
Perl and Raku Programming
D.perl
A brief introduction to the 2021 virtual FOSDEM devroom, talk overview, code of conduct and Community Affairs Team
2/6/21
Perl and Raku Programming
Elizabeth Mattijsen
D.perl
Set operators are often seen as a way to handle (semi-)complex mathematical issues, the only exists in Set objects. But set operators are so much more! This presentation will show you how to make use of set operators in your day-to-day code, making your code more concise, better readable and better maintainable.
2/6/21
Perl and Raku Programming
Curtis Poe
D.perl
I plan to bring modern OO to the Perl core. Modern enough that it leapfrogs the capabilities of the OO systems of many other dynamic languages. I’ve been stealing ideas from Stevan Little, Damian Conway, and anyone else foolish enough to leave their ideas lying around. I have no pride. Sawyer’s expressed interest and it's likely it will go into the Perl core, though with the upcoming Perl governance changes, the timeline is unclear. I’m not going to beat around the bush: writing ...
2/6/21
Perl and Raku Programming
Nigel Hamilton
D.perl
The TPF is passionate about helping our software communities flourish. This is an update from a legal and marketing perspective on the communities' brands and some suggested next steps to help them flourish.
2/6/21
Perl and Raku Programming
Curtis Poe
D.perl
Raku is an amazing programming language, but many examples online are very complex and can be hard to follow. This talk highlights some amazing features of Raku, but assumes zero knowledge of the language.
2/6/21
Perl and Raku Programming
Nigel Hamilton
D.perl
Spending a short time working as a barrister opened my eyes to a profession that is hundreds of years old. In comparison to the legal profession, the computing profession has barely got started! This talk is a mash up of when these two worlds collide. As professional Perl and Raku programmers are there any lessons we can borrow from the arcane world of barristers? Which practices could help power Perl and Raku for the next hundred years?
2/6/21
Perl and Raku Programming
Paul Evans
D.perl
When Perl gained the ability to support custom keywords provided by modules it started down the path that CPAN modules would experiment with new language ideas. Already a number of such modules exist, and it is likely this idea will continue to develop. What new ideas might turn up in the next few years, and will any of them evolve to become parts of the actual core language?