Jupyter Notebook is an app for data analysis. The idea is to combine documentation and the code! My wife uses it for her data science courses from Coursera. Once she complained that some tasks took whole night to complete on her laptop. Her Sony Vaio is pretty powerful, but definitely not a mainframe. When I noticed that Notebook is actually a web application I immediately suggested to run it in Amazon AWS! This is a short instruction how to setup Jupyter Notebook there.
First you have to provision EC2 instance with Amazon Linux. I recommend so called “compute-optimized” instance types (cX) as they provide max CPU power. Amazon Linux already comes with Python 2.7.12 which is enough for Jupyter. Installing Jupyter is pretty simple:
sudo pip install jupyter
Then you need to start it. Here is what I do:
ssh -i <rsa-key> ec2-user@<ec2-machine-public-dns>
jupyter notebook --no-browser
First I login to the EC2 instance. Then I start screen session so I can easily logout/disconnect and let jupyter run in background. Third line is launching Jupyter Notebook. Note “no-browser” that’s because by default Notebook would try launching browser and we don’t want that. Jupyter will print out login URL similar to http://localhost:8888/?token=a917d6207a4726774e2fd4d6053d12e24b0326628e2d7350. Copy it to you clipboard.
Next step is to create an SSH tunnel to access our Jupyter instance:
ssh -i <rsa-key> -fNL 8888:localhost:8888 ec2-user@<ec2-machine-public-dns>
Now you can open you browser and pasted saved URL:
The last thing you can do (if you want to try data science staff) is installing popular Python packages. But before that you need to install GCC and its prerequisites. In Amazon Linux (and Red Hat) it’s super easy:
sudo yum groupinstall "Development Tools"
Then you can install actual packages using pip:
sudo pip install numpy
sudo pip install pandas
sudo pip install xgboost
sudo pip install sklearn
And so on…
We’re all used to package managers in Linux distros: Aptitude in Debian, yum in Read Hat, emerge in Gentoo. This list can be continued. I was very surprised when my colleague suggested to install Homebrew on Macbook to get some tools. I did that and I’m happy. Almost all my tools now get installed with brew install.
If I’m stuck with Windows on my home laptop I thought maybe there is a package manager for Windows? Yes! There is! It’s called Chocolatey! Looks like it’s based on Power Shell so the installation process is super easy. Plus reading there guide is enough to make it work.
As of now I have installed bunch of my tools using choco install: JDK, Groovy, Gradle, Maven and Intellij IDEA! Isn’t that neat?
The first time in my carrier I had to access SFTP from a server w/o direct access to Internet! Sure thing I had to use proxy. But it took me a while to find a workable solution:
sftp -oProxyCommand="nc -X connect -x your_proxy:port %h %p" username@hostname
Google suggests a lot of strange tools like proxy-connect, connect etc. But nothing of that is available in CentOS! For the sake of truth I should mention that proxy has to be configured correctly. In my company Ops guys use Squid which by default forbids everything except HTTP/HTTPS! So they had to tune it a bit to allow port 22.
PS. One guy told me I could use SSH as SOCKS proxy! I found this interesting article describing that in details…
4 years ago I got Dell Precision 6400 at my previous job. It’s BIOS is dated 2011. At that time it was a very descent laptop: Intel i7, 8G of RAM, 256Gb SSD disk and 1Gb ATI Radeon 6740M! It came with Windows 7, but I replaced it with Linux Mint (I even had Gentoo for a short time). I liked it and got used to it despite its enormous size and weight. At some point a year ago I realized I need something more mobile and company gave me Lenovo T460 which I replaced with 15″ MacBook Pro eventually.
Few days ago I felt like I have to use separate workstation for my toy projects so I won’t interfere with my work environment and settings. I don’t want to use my gaming desktop and I want some mobility. So I decided to revive my Dell.
You know I had no troubles with Linux Mint, but I like trying something new. I picked Fedora 24 in memories of my first Linux desktop running Red Hat 9 ten years ago. But I didn’t like Gnome3 and GUI was pretty slow. Eclipse was almost unusable with ugly jitters when you scroll your code. Unfortunately the same thing happened with Linux Mint 18 Sarah! While I remembered Cinnamon as a fast window manager Eclipse Neon still was slow. Plus I had troubles with WiFi card! Sure my GPU driver was the real root cause! As I found out AMD stopped supporting drivers for X11 and latest XOrg server can’t use them. So I ended up with built-in open source driver which kinda works, but looks like can’t use all the power of ATI GPU (for instance, glxgears showed only 60 fps). I’ve heard that AMD (and NVidia) are working hard to provide there native version of open source driver (or at least partly opened), but they are targeting only latest GPUs 🙁
I was disappointed and almost gave up on my old buddy. And then yesterday I decided to try Windows 10. Just for fun. Apparently you can download Windows ISO for free directly from Microsoft site. You don’t need any product keys and you will get fully functional OS (with reminders to activate it). I made a bootable USB stick and installed Windows 10 Home. And you know what? I am impressed how flawlessly it went and how responsive Eclipse Neon is now! I even tried 3dMark tests and Vantage test showed my old horse is very close to gaming laptop for DX10! I’m not going to play games, but looks like I will stay with Windows for now. Until I got something more modern…
At the end is a quick reminder how to make a bootable USB stick in Linux. So basically there are two methods. For Linux distros usually it’s enough to use dd:
dd if=linux.iso of=/dev/<usb_stick_dev> bs=4M
For Windows you have to use fdisk (or parted) for creating NTFS partition first. (Don’t forget to mark it as bootable!) And then copy all files from Windows ISO image to that USB stick partition. And that’s it!
Yesterday I checked my blog and got “Request timed out”. As you can guess from the title I become a victim of XML RPC exploit. There a lot of info on Internet describing what XML RPC exploit is and how to defend your blog. I will describe how I fought that attack myself. Well, with the help from mighty Google search 🙂
So when I logged into my AWS instance the first symptom was high CPU load from httpd. Which is not very surprising for t2.micro instance type 🙂 Then I checked /var/log/httpd/access_log and found tons of events like this:
220.127.116.11 - - [14/Oct/2016:20:03:56 +0000] "POST /xmlrpc.php HTTP/1.0" 500 251 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
The first mitigation is to disable access to /xmlrpc.php in .htaccess:
Redirect 301 /xmlrpc.php http://127.0.0.1
That reflects the attacker and unloads your server so you can log in to WordPress admin console. The next thing is to shield your WordPress from similar attacks.
For that I installed WP Fail2Ban Redux plugin which logs all malicious events (including xmlrpc) to system log so they can be analyzed by Fail2ban service. Then I installed actual fail2ban service using yum and copied configurations from plugin’s folder. Note that you have to specify correct path to system log file plus default configuration does not actually ban (in Amazon Linux at least). Here is my local.jail for WordPress:
enabled = true
filter = wordpress-hard
logpath = /var/log/messages
maxretry = 2
action = iptables-multiport[name=WordPress, port="http,https", blocktype=DROP]
enabled = true
filter = wordpress-soft
logpath = /var/log/messages
maxretry = 5
action = hostsdeny
Basically these rules will block furious attacker using firewall (by dropping tcp packets). The wordpress-soft rule is about password attack and it just adds host to the hosts.deny for 10 minutes (default ban time). After that you can remove redirect rule from .htaccess if you need xmlrpc feature. I will keep it disabled…
I haven’t visited my blog for a while, so was very surprised to see “Error Establishing a Database Connection” page. At first I thought someone had hacked my box 🙂 Instead MySQL server was down. I checked server logs and found fatal errors from mysql:
InnoDB: Fatal error: cannot allocate memory for the buffer pool
I googled that issue immediately and according to StackExchange my database couldn’t allocate more system memory and process died. Hm, 1G of RAM is not enough for a tiny blog???
Anyway, I have moved WordPress database to Amazon RDS. I went with t2.micro so it shouldn’t cost much. For the safety reason my instance does not have public IP and allows only connections from WordPress host.
I installed my WordPress blog only 7 days ago. I never shared the link cause it’s kinda my private experiment for fun. But it’s been a second day someone is atacking my website. It’s a simple attack – password guessing– and there is no chance for attacker to succeed as I’m not there old granny and use randomly generated passwords. Anyway it’s exciting to be part of this cruel world!
Sucuri plugin logs failed logins, so I’m able to see there IP-address (well, gateway). According to ip-www.net it’s Russian Federation. Saint Petersburg City if to be precise. Wow! Rumors don’t lie that russian hackers are everywhere 😀
The easiest thing would be to block there access to my IP. But AWS security groups are always permissive. So the only cheap way is to drop there packets using iptables:
sudo iptables -I INPUT -s 18.104.22.168 -j DROP
I googled help on StackOverflow 🙂
If you want to restore your firewall rules during reboot, then don’t forget to call iptables-save!
I am not a security guru nor a hacker. But I listen to what people say about security, vulnerabilities, exploits etc. And I am aware that default installation of anything (including whole LAMP stack) is not secured.
Amazon’s guides about LAMP and WordPress contain some security topics. So your installation will not be completely ridiculous 🙂 But Amazon also has a nice guide for setting up SSL/TLS. And it worked perfectly for me! I followed everything step by step, got free SSL certificate from startssl.com and even tested my server using Qualys SSL Lab! Whoa!
I’ve also applied security guide from WPBeginner and installed security plugin to monitor my blog and alert me if something goes wrong! Interesting stuff!
That idea with AWS VPN doesn’t work in long term. My client couldn’t connect again. So I dropped that stack and forgot about VPNs for now.
Well, that CloudFormation template at least can be used if someone needs to provision temporal personal VPN in few minutes.
Ok, I’ve got some feedback after using solution I described previously.
First of all that thing stopped working the very next day. Given that I didn’t have SSH keys to login there, I couldn’t investigate why. I just dropped CloudFormation stack and created another one. So far so good.
Second disappointment is that I couldn’t make it work with Linux Mint. Network Manager’s PPTP plugin doesn’t work at all. I even installed client for L2TP/IPSec, but failed to get working connection. I download there CloudFormation template and found out there is a full PPTP/L2TP/IPSec configuration there. So probably investigating that carefully I may come up with correct configuration for L2TP/IPSec client.
Mac OS X works flawlessly though. And I bet Windows 10 will work too…