[MAC][IDE][ATOM] build python3.4 on atom editor

1.download and installed the atom in mac as usual way.

2.download the package atom-runner
(from the atom menu->preference->install...)
ref:https://github.com/lsegal/atom-runner
3.go to atom->config

you will open below setting, and add the red line into it. (careful with indent.)

"*":
  "atom-runner": {}
  core:
    automaticallyUpdate: false
    destroyEmptyPanes: false
    openEmptyEditorOnStart: false
    themes: [
      "one-dark-ui"
      "atom-dark-syntax"
    ]
  editor:
    fontSize: 18
    invisibles: {}
    showIndentGuide: true
    showInvisibles: true
    zoomFontWhenCtrlScrolling: true
  "exception-reporting":
    userId: "d4912b62-aa97-a2d1-4abc-b19393b94058"
  runner:
    scopes:
      python: "/Library/Frameworks/Python.framework/Versions/3.4/bin/python3"
  welcome:
    showOnStartup: false



4. you can change the keymap if you want.

[rubyonrails] call out python

This method is not running command in background.


put your test.py under
/app/assets/pythonscripts/test.py

import os, sys
import time

time.sleep(5)
print("python in rails ha ha 5  ")




in show.html.erb

<% cmd = "python3.4 "+ "#{Rails.root}/app/assets/pythonscripts/test.py" %>
<% result = %x[#{cmd} &] %>
<%= result %>

[Sublime text ] support big5

簡體中文GBK
“Preferences”-->“Package Control”,或者使用快捷键CTRL+SHIFT+P调出


在输入框内输入“Install Package”，点击下方自动提示出来的“Install Package”

在输入框内输入“CovertToUTF8”，下面就会相应的插件，鼠标点击安装。

插件就会安装成功，无需重启Sublime Text,你就会发现中文内容已经能够正确显示了。


Big5
Again, but this time, install Codecs33
Reopen sublime

[R][MAC]

1.When install package always shows

tar: Failed to set default locale 

2.Open Mac terminal:

defaults write org.R-project.R force.LANG en_US.UTF-8

3.Restart R

Done.

[git] set different user.name user.email in different repository

Go the the folder of your local repository

and type

git config user.name "xxxxxx"

git config user.email "xxxxx@www.com"

take a check 

cd .git

cat config

[MAC] Terminal hostname not mine.

it appears other's name who give you DNS.

go to terminal and check

xxxx $ hostname
xxxxxx

if the xxxx

$ sudo scutil --set HostName "xxxx's MacBook Air"

QUIT terminal, reopen again.

[octave] install octave 4.0.3 under mac osx EI 10.11, plot OK

ref: https://kiskeyix.org/articles/605

1.
install hombrew
install AquaTerm from sourceforge
brew tap homebrew/science
brew reinstall gnuplot --with-aquaterm
gnuplot # make sure it says "terminal set to aqua"
brew install lua51 # yes, you also need this old version of Lua
brew install octave

2.
vim ~/.octaverc
add below....

octave:1> graphics_toolkit('gnuplot')
octave:1> setenv('GNUTERM','aqua')

check again

octave:1> getenv("GNUTERM")
ans = aqua
octave:2>



=========================
For waring message when you plot
warning: could not match any font: *-normal-normal-10


use...

brew uninstall fontconfig

brew install fontconfig --universal

OK

[ML] logistic gradient accent and Andrew Ng gradient decent

ref:

http://www.csdn123.com/html/topnews201408/66/10366.htm


the gradient fun is l(n)

normally hope probability biggest, so use gradient Accent to find largest theta of l(n)

but

Andrew Ng set J(theta)=-(1/m)l(n), there has a "-", so become find the minimum , so
use gradient decent.

[python3.x] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position

Can't fix by root cause, but I try another way, reedit the file you hit.

1. open that file in vim, and you can see the  <0x92> in the file, remove and save. Done.
2. simply open file in sublime text 3 or TextEdit something, find some strange mark. remove it. Done.

ps. if you wondering what is the encoding of your file, go to sublime , show console and
     type-> view.encoding()

[mac] add path for adb,fastboot,.... to PATH

get mac version of adb , fastboot....

=> download Android Studio from google developer site.


copy those bin to folder whether you want.  ex: /Users/xxxxx/Documents/bin

and

vim ~/.bash_profile

PATH="/Library/Frameworks/Python.framework/Versions/3.4/bin:/Users/xxxxx/Documents/bin:${PATH}"

export PATH

save , and reopen the terminal.

Done.

[mac] Connect FTP from mac to ubuntu by vsftpd

[ubuntu side]

sudo apt-get install vsftpd



[mac side]

download filezilla client.

create a new ftp , use account and password to login.



ref:
http://yunol.com.tw/phpbb3/viewtopic.php?f=15&t=25132

[IDE]Sublime text 3 + Anconda ST3 => Autocomplete pandas dataframe setting

add following in to

Preferences->Setting-User

{

"auto_complete_triggers":
[
   {
       "characters": ".",
       "selector": "source"
   }
],
}

Preferences -> Package setting->Anconda->Setting-User

add at the tail.

{


    "anaconda_linting": false,
    "python_interpreter": "/Library/Frameworks/Python.framework/Versions/3.4/bin/python3",
    "auto_python_builder_enabled": false,
}

[IDE] Sublime text 3 + Anconda ,autocomplete quickly config

Sublime Text → Preferences → Settings—User

"auto_complete_triggers":
[
    {
        "characters": ".",
        "selector": "source"
    }
],

[MacOS] install python3.4 pandas , request, matplotlib

pip3.4 install pandas
pip3.4 install request
pip3.4 install matplotlib



SUPER EASY.....

[Coursera] Backup method. python3.4

ref: https://github.com/coursera-dl/coursera-dl
ref:http://www.owstartup.com/2016/06/17/how_to_download_coursera_courses_before_they_are_gone/


start:

1.
pip3 install coursera-dl

2.
go to folder which you want to download

3.
get url of the course you want to download . ex: https://class.coursera.org/prob-001

4. type
coursera-dl -u xxxxxxx@xxxx.com -p xxxxxx prob-001

5. start downloading...

[python] auth with NTLM server & install python3.4 with requests_ntlm

Install requests_ntlm



1. need install request
2. install python-ntlm3
=>by download source code,

go to the folder you extracted files.
and use command

"sudo python3.4 setup.py install"

to install it.

3.install requests_ntlm
source code:https://github.com/requests/requests-ntlm

go to the folder you extracted files.
and the same, use command 
"sudo python3.4 setup.py install"


Done.

 ----------------------------------------------------------------------------

    from requests_ntlm import HttpNtlmAuth

    rs_aps = requests.session()
    url='http://xxxxx.xxxx.com/'
    res = rs_aps.get(url, auth=HttpNtlmAuth('domain\\account', 'passwd'))
    res.encoding='big5'
    #print(res.text)
    #get cookies
    #cookie will change every time
    for c in res.cookies:
        ASP_Cookies_name=c.name
        ASP_Cokies_value=c.value
        print(ASP_Cookies_name, ASP_Cokies_value)

[Machine Learning] 'Training/Cross-Validation/Test

from:http://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set

The concept of 'Training/Cross-Validation/Test' Data Sets is as simple as this. When you have a large data set, it's recommended to split it into 3 parts:

++Training set (60% of the original data set): This is used to build up our prediction algorithm. Our algorithm tries to tune itself to the quirks of the training data sets. In this phase we usually create multiple algorithms in order to compare their performances during the Cross-Validation Phase.

++Cross-Validation set (20% of the original data set): This data set is used to compare the performances of the prediction algorithms that were created based on the training set. We choose the algorithm that has the best performance.

++Test set (20% of the original data set): Now we have chosen our preferred prediction algorithm but we don't know yet how it's going to perform on completely unseen real-world data. So, we apply our chosen prediction algorithm on our test set in order to see how it's going to perform so we can have an idea about our algorithm's performance on unseen data.

Notes:

-It's very important to keep in mind that skipping the test phase is not recommended, because the algorithm that performed well during the cross-validation phase doesn't really mean that it's truly the best one, because the algorithms are compared based on the cross-validation set and its quirks and noises...

-During the Test Phase, the purpose is to see how our final model is going to deal in the wild, so in case its performance is very poor we should repeat the whole process starting from the Training Phase.

[Machine Learing] Training set 6: cross validation set 2: testing set 2

我们首先定义train、cv（cross validation）和test error。

之后我们根据train error训练模型。如果是模型比较时，我们就训练出多个不同的模型，如上图。

之后，对于模型选择。我们让cross validation set通过的这些模型，并计算这些模型的error（cv error）。我们选cv error最小的那个模型作为最好的模型。

最后，对于估计模型的performance（也就是generalization）：我们让   test set通过我们刚刚用cv error选出的模型，得到test error，作为performance。

需要强调的是，如果不做模型选择，那么就不需要validation set了。只要training和test set。两者的比例大概是7:3。

然后，这里解释下为什么做模型选择要validation set和test set分开。首先我们不能拿training set做模型选择，因为那样没有generalization了（我们的算法就是根据training set来优化模型的，所以没有再用这个set做模型选择没意义）。其次，如果我们用test set选模型，那么选择的模型就fit到 test set，结果我们同样失去了generalization。所以要用validation set选模型。用test set测generalization。

=>

Use cross validation set to chose polynomial model (theta from training set), use testing set data to judge the performance of model which you just get. 

****

If you do not have to select model, then just use training set 7 : testing set 3 .

[octave] element by element operation

      plus       .+
      minus     .-
      times     .*
      rdivide   ./
      ldivide   .\
      power     .^  .**

https://www.gnu.org/software/octave/doc/v4.0.0/Arithmetic-Ops.html#Arithmetic-Ops

x .+ y

    Element-by-element addition. This operator is equivalent to +.


x = [1 2 3;
     4 5 6;
     7 8 9]

y = [1 1 1;
     1 1 1;
     1 1 1]

x.+y # equal x+y, because + originally is a element by element operator

ans =

    2    3    4
    5    6    7
    8    9   10



x .* y

    Element-by-element multiplication. If both operands are matrices, the number of rows and columns must both agree, or they must be broadcastable to the same shape.

x = [1 2 3;
     4 5 6;
     7 8 9]

y = [10 2 1;
     10 2 1;
     10 2 1]

x*y #normal *

ans =

    60    12     6
   150    30    15
   240    48    24

x.*y # element by element 1*10 4*10 7*10 .....
ans =

   10    4    3
   40   10    6
   70   16    9



x ./ y

    Element-by-element right division.



x = [1 2 3;
     4 5 6;
     7 8 9]

y = [10 2 1;
     10 2 1;
     10 2 1]

x./y
ans =

   0.10000   1.00000   3.00000
   0.40000   2.50000   6.00000
   0.70000   4.00000   9.00000




y = [10 2 1;
     10 2 1;
     10 2 1]
m=5
y./5 # equal y/5

ans =

   2.00000   0.40000   0.20000
   2.00000   0.40000   0.20000
   2.00000   0.40000   0.20000

[octave]broadcasting

https://www.gnu.org/software/octave/doc/v4.0.0/Broadcasting.html#Broadcasting


在octave 中, broadcasting 的意思是
當兩個唯度不同矩陣要運算時, 維度小的 會自己擴充成跟維度大的一樣維度之後, 才進行運算


ex:

x = [1 2 3;
     4 5 6;
     7 8 9]

y = [10 20 30]

x+y  # 這本來是不能作的, 因為x y 維度不同, 但他自動幫你變成

y = [10 20 30
     10 20 30
     10 20 30];

這就是broadcasting, 此時 y 維度就跟x 一樣, 可以作加法了

>>output
warning: operator +: automatic broadcasting operation applied
ans =

   11   22   33
   14   25   36
   17   28   39


(他也會出現警告, 提醒你這個+法經過了broadcasting)



 如果兩個matrix 一樣dimension , 則直接就是
element by element 相加



x = [1 2 3;
     4 5 6;
     7 8 9]

y = [1 2 3;
     4 5 6;
     7 8 9]

x+y
ans =

    2    4    6
    8   10   12
   14   16   18

[octave ] octave zeros function

Built-in Function: zeros (n)
Built-in Function: zeros (m, n)
Built-in Function: zeros (m, n, k, …)
Built-in Function: zeros ([m n …])
Built-in Function: zeros (…, class)

    Return a matrix or N-dimensional array whose elements are all 0.

    If invoked with a single scalar integer argument, return a square NxN matrix.

    If invoked with two or more scalar integer arguments, or a vector of integer values, return an array with the given dimensions.

    The optional argument class specifies the class of the return array and defaults to double. For example:

    val = zeros (m,n, "uint8")


a=[1, 2, 33;4 ,5, 6; 7 ,8, 66;55 ,476, 22]
a =

     1     2    33
     4     5     6
     7     8    66
    55   476    22

[rows columns]=size(a);
rows =  4
columns =  3

zeros (rows,columns, "uint8")

ans =

  0  0  0
  0  0  0
  0  0  0
  0  0  0


zeros(4) # will give you 4x4 matrix
ans =

   0   0   0   0
   0   0   0   0
   0   0   0   0
   0   0   0   0


zeros(4,1) #want a 4x1 zero vector
ans =

   0
   0
   0
   0

[octave] size function

size function

https://www.gnu.org/software/octave/doc/v4.0.1/Object-Sizes.html#XREFsize


Built-in Function: size (a)
Built-in Function: size (a, dim)

    Return the number of rows and columns of a.

    With one input argument and one output argument, the result is returned in a row vector. If there are multiple output arguments, the number of rows is assigned to the first, and the number of columns to the second, etc. For example:

    size ([1, 2; 3, 4; 5, 6])
       ⇒ [ 3, 2 ]

    [nr, nc] = size ([1, 2; 3, 4; 5, 6])
        ⇒ nr = 3
        ⇒ nc = 2

    If given a second argument, size will return the size of the corresponding dimension. For example,

    size ([1, 2; 3, 4; 5, 6], 2)
        ⇒ 2

    returns the number of columns in the given matrix.

    
Testing:
    
a=[1, 2, 33;4 ,5, 6; 7 ,8, 66;55 ,476, 22]
a =

     1     2    33
     4     5     6
     7     8    66
    55   476    22

[rows columns]=size(a); 
rows =  4
columns =  3


size(a,1) # 1 means get rows ;  2 means get columns
ans =  4


size(a,2)
ans =  3



-------------------------------------------------------------------------------









 

[Octave] Octave matrix slice

CASE1:

a=[1, 2, 33;4 ,5, 6; 7 ,8, 66];

a =

    1    2   33
    4    5    6
    7    8   66

a(3)       # result is a scalar
 ans =  7


a(1:4)     # show range form index 1 to(:) index 4, and result is a row vector
ans =

   1   4   7   2

a([1; 9])  # show range form index 1 and index 9 result is a column vector
ans =

    1
   66

a(1, [1, 3])  # row 1, columns 1 and 3
ans =

    1   33


a(3, 1:3)     # row 3, columns in range 1-2
ans =

    7    8   66

a(1, :)       # row 1, all columns , use really often !!!

ans =

    1    2   33


Case2:

a = [1, 2, 3, 4]
a =

   1   2   3   4

a(1:end/2)        # first half of a => [1, 2],end is the last element in the matrix
ans =

   1   2


a(end + 1) =5   # append element
 a =

   1   2   3   4   5

a(end) = []      # delete element
a =

   1   2   3   4

a(1:2:end)        # odd elements of a => [1, 3]
ans =

   1   3

a(2:2:end)        # even elements of a => [2, 4]
ans =

   2   4

a(end:-1:1)       # reversal of a => [4, 3, 2 , 1]
ans =

   4   3   2   1



 CASE3: often use in machine learning cost function

a=[1, 2, 33;4 ,5, 6; 7 ,8, 66];

a =

    1    2   33
    4    5    6
    7    8   66

[99 ; a(1,:)']

 ans =

   99
    1
    2
   33


num_labels=4;
zeros(num_labels,1)

ans =

   0
   0
   0
   0


a=[1, 2, 33;4 ,5, 6; 7 ,8, 66;55 ,476, 22]
a =

     1     2    33
     4     5     6
     7     8    66
    55   476    22

a(:,2:end) #from second column  to end

ans =

     2    33
     5     6
     8    66
   476    22

a(1,:) #get first row
ans =

    1    2   33

a(2:end) # start from element 2 to end

ans =

     4     7    55     2     5     8   476    33     6    66    22

a(:) # unroll every element


ans =

     1
     4
     7
    55
     2
     5
     8
   476
    33
     6
    66

[python] install pandas on ubuntu 14.04 python3.x

sudo apt-get install python3-pandas

[octave] build octave under sublime text 3 (pause not work)

tools->build system->new build system

    {
        "path": "/usr/bin/",//path of your Octave bin folder
        "cmd": ["octave", "--no-site-file", "-p $file_path", "$file_name"]
    }

[python] install matplotlib in python3.x under ubuntu 14.04

sudo apt-get install python3-matplotlib

[python] install numpy under ubuntu 14.04, python3.x

sudo apt-get install python3-numpy

[ubuntu] screenshot area auto save to desktop by shutter ; ask to save but default folder is Desktop by default screeshot app.

1. For original default Ubuntu screenshot app

a.Just install dconf-editor:

sudo apt-get install dconf-editor

b.Open dconf-editor and edit the

auto-save-directory field org > gnome > gnome-screenshot
=>/home/xxxx/Desktop



Ps. shortcut [shift+PrintScrn]  is defaut shortcut for screenshot of area




2. install app shutter

by....

http://apt.ubuntu.com/p/shutter


after configure all setting....


 you need to let original PrintScrn key to call shutter instead the original

screenshot app.


Go to setting -> keyboard -> go down to add custom shortcut
and add new one.

Use the command =>
    Full screen with Shutter – shutter -f
    Take a screenshot of an area with Shutter – shutter -s

[Machine Learning][Study Note][Week1] Matrix

可以利用矩陣運算, 將同一組資料對於不同多個h(x)一次帶入,非常方便

 

一些其他特性

矩陣 I 如果為Identity, A*I=I*A, 則有交換律

否則一般矩陣 A*B != B*A

矩陣有結合律

A+(B+C)=(A+B)+C

可以用軟體求Inverse

A*A(inverse)=I

ex: 3*1/3=1, 則1/3就是 3 的inverse.  note: 0 沒有 inverse.

[Machine Learning][Study Note][Week1] Cost Function & Gradient Descent

一個線性回歸的公式

h(x)=θ0+θ1*x

h(x)=hypothesis



 求 Cost Function 的最小值, 也就是說, 要對 h(x) 找cost 最小的,此h(x) 就會當作

我們的回歸公式

所以, 我們要決定怎樣的東西來當作cost function? 這裡我們用距離的平方,
所以, 找一個一階線性回歸的最佳解, 就是這條線能讓所有訓練資料帶進去
得到所有人距離這條線的總和最小, 那就把係數θ0, θ1拿來用.

Gradient Descent
＝你把普通的兩變數函數畫出來後, X=θ0,Y=θ1,Z=Cost,
你就想像你起始站在山頂上, 但這坐山可能不是最高的山, 然後一步一步的
往山下走, 直到最低點(local min), 所以對於一般的函數來說, 由於
可能有另外一座高山,導致也得到不同的local min2, 因此這這個演算法
並沒有得到全局最min 解. (我們可以控制走得步伐)


所以, 有了cost function 後, 我們要對他求min, 利用Gradient Descent 演算法
來求,

在若在2變數 θ0, θ1 之下, 用軟體畫出所有可能的Y=h(x), 會是一個3D
圖, 而剛好是個凸邊型, 也就是弓型, 而雖然 Gradient Descent 會求出
local min 值, 但是因為這個 cost function 是凸邊型, 所以他的local min
就是全局最小, 所以就得到我們要的答案了.


事實上 梯度下降法能用在所有地方, 不只是線性回歸.
後面還有正規方程 normal equation 也能解最小值.

[Machine Learning][Study Note][Week1] Supervised / Unsupervised Learning

Supervised Learning
=由一些已經知道解答的數據群中, 訓練出模型的方式

這主要分兩大類 1.回歸 2.分群

1.回歸=用於連續值得輸出預測

譬如你有一堆這個區域房子成交價
X= 坪數
Y=售價

你用這些歷史數據訓練出一個回歸模型, 於是你可以利用這個回歸模型去預測新的X

譬如你朋友想要在這區域賣房子, 於是他給你他房子的坪數, 你套入那個練出來的模型

得到Y, 則可以建議他以Y價格售出



2. 分群=用於離散值得輸出預測

譬如

有一堆有腫瘤病患的資料, 而假設我們X只看一個feature 叫做腫瘤直徑
(實際上會有很多feature)

而 Y就是 0=不是惡性腫瘤 1=是惡性腫瘤

所以把歷史資料都畫出來之後, 訓練出一個演算法可以分群, 以便之後出入新的病人的
腫瘤直徑, 可以預測是否為惡性


又

X=腫瘤直徑 , Y=年齡

一樣可以訓練出分群模型








Unsupervised Learning
=這是說你不知道這推數據姿料理有怎樣的規則, 你想要用演算法看看能不能找出這些資料中存在某種結構(可能這些data能分成不同群)

譬如說
每天看到的新聞網頁, 他們去其他網路上收集新聞之後, 他會被自動分到同一類, 譬如某些新聞會被分成運動, 財金 , 這就是跑了演算法之後, 發現能被分類


又或者你有一大堆的健身客戶資料, 你不知道客戶之間有甚關係, 去跑之後, 發現
這些客戶之中, 又有很多膝蓋受過傷的細分市場客戶

簡單來說,
我手上有一堆數據, 我根本不知道裡面有甚東西, 我沒給演算法答案,
請問你（演算法）你可以幫我找出這些數據當中的類型嗎？
我只告訴你演算法, 沒告訴你這些資料的答案(譬如這個人該分到哪群)

[python] debug log with logging config, pprint with logging, logging to file and still output at stdout

import pprint
import logging
from source.projectparameter import project_parameter as pp


def logconfig():
    print_format="%(asctime)s,line:(%(lineno)d): %(message)s"

    logging.basicConfig(filename="abspath/xxxx_debug.log",
                        filemode='a',
                        format=print_format,
                        datefmt='%H:%M:%S',
                        level=logging.DEBUG)
    stderrLogger=logging.StreamHandler()
    stderrLogger.setFormatter(logging.Formatter(print_format))
    logging.getLogger().addHandler(stderrLogger)
    return 0



def show_all_init_paramater():

    for tag in pp:
        #print('\n',tag,'with',len(pp[tag]),'items')
        #logging.debug("\n",tag,"with",len(pp[tag]),"items")
        logging.debug("%s with %s items",tag,len(pp[tag]))
        logging.debug("\n"+pprint.pformat(pp[tag]))
    return 0


def main():
    logconfig()
    logging.debug('=======NEW SECTION======')
    show_all_init_paramater()


    return

main()

[python] Ubuntu, python 3.x list files under folder , list modify time , create time of files. 列出資料夾下的檔案list, 和修改時間或其他屬性

import os,time



file_list=os.listdir("/home/xxxxxx/Desktop")
#print(file_list)


for file in file_list:
    if '~'  in file:
        file_list.remove(file)

print(file_list)

for file in file_list:
    (mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime) = os.stat(file)
    print (file,"\nlast modified: %s" % time.ctime(mtime),'\nCreate time   ',time.ctime(ctime),'\n')

[python] python 3.x , date itreating from start date to stop(today) date.

from datetime import datetime,timedelta


def daterange(begin, end, delta = timedelta(1)):
    """Form a range of dates and iterate over them.

    Arguments:
    begin -- a date (or datetime) object; the beginning of the range.
    end   -- a date (or datetime) object; the end of the range.
    delta -- (optional) a timedelta object; how much to step each iteration.
             Default step is 1 day.

    Usage:

    """
    if not isinstance(delta, timedelta):
        delta = timedelta(delta)

    ZERO = timedelta(0)

    if begin < end:
        if delta <= ZERO:
            raise StopIteration
        test = end.__gt__
    else:
        if delta >= ZERO:
            raise StopIteration
        test = end.__lt__

    while test(begin):
        yield begin
        begin += delta


def main():

    start = datetime(2010, 1, 1,19,00,00)
    stop = datetime.today()
    for x in daterange(start,stop):
        # only when today, it need to download after 7:00pm, so force the start date all is 19:00:00
        if(stop > x):
            #print(stop,x)
            year=str(x.year)
            month=str('%02d' % (x.month))
            day=str('%02d' % (x.day))

            download_xxxxx_from_web(year,month,day)

        else:
            print('For ',x,'Need to wait to 7:00')
    return


main()

[ubuntu] enable ssh login

sudo apt-get install openssh-server

[Ubuntu 14.04.1] 解決Chrome/ Firefox字體變成標楷體或者新細明體的問題

[Ubuntu 14.04.1] 解決Chrome/ Firefox字體變成標楷體或者新細明體的問題

fonts-arphic-ukai,fonts-arphic-uming
這兩個套件

移除他即可

# sudo apt-get remove fonts-arphic-ukai && sudo apt-get remove fonts-arphic-uming



***** logout, login again

[ubuntu] 14.04 install 酷音

1.安裝酷音

sudo apt-get install ibus-chewing

2.

logout, login again, go to text entry , and add

Chinese (Chewing)

 

[python] python numpy static , sum, mean, std (axis = dimension)

a = np.array([[1, 2], [3, 4],[5, 6]])

print('-------------------')
print(np.sum(a))
print(a.sum())
print(a.sum(0))
print(np.sum(a,axis=0))
print(np.sum(a,axis=1))
print('-------------------')
print(np.mean(a))
print(a.mean())
print(a.mean(0))
print(np.mean(a, axis=0))
print(np.mean(a, axis=1))

print('-------------------')
print(np.std(a))
print(a.std())
print(a.std(1))
print(np.std(a, axis=0))
print(np.std(a, axis=1))

output:

-------------------
21
21
[ 9 12]
[ 9 12]
[ 3  7 11]
-------------------
3.5
3.5
[ 3.  4.]
[ 3.  4.]
[ 1.5  3.5  5.5]
-------------------
1.70782512766
1.70782512766
[ 0.5  0.5  0.5]
[ 1.63299316  1.63299316]
[ 0.5  0.5  0.5]
[Finished in 0.1s]

ps: axis 0 means do operator in each column individual. (first dimension.) 

print(np.sum(a,axis=0))

[1,2]

[3,4]

[5,6] 

=[1+3+5 , 2+4+6]= [9,12]

axis 1 means to operator second dimension.  

print(np.sum(a,axis=1)) 

=[1+2,3+4,5+6]=[3,7,11]

---------------------

a = np.array([[1, 3,3, 4, 5, 6],[1, 3,3, 4, 5, 6]])
b = np.array([[1, 2,3, 4, 5, 6],[1, 1,1, 1, 1, 1]])
print(a)
print(np.mean(b, axis=0))
print(a-np.mean(b, axis=0))

[[1 3 3 4 5 6]
 [1 3 3 4 5 6]]

[ 1.   1.5  2.   2.5  3.   3.5]

[[ 0.   1.5  1.   1.5  2.   2.5]
 [ 0.   1.5  1.   1.5  2.   2.5]]
[Finished in 0.1s 

[python] python 3.4 Ubuntu 14.04 install scikit-learn by building from source code.

0.Optional, If you needed....

sudo apt-get install build-essential python3-dev python3-setuptools \
                     python3-numpy python3-scipy \
                     libatlas-dev libatlas3gf-base

1. download scikit-learn source code form

https://pypi.python.org/pypi/scikit-learn/0.15.2#downloads


The same as below, download the first one, scikit-learn-0.15.2.tar.gz

and extract it in your downloaded folder.



2. go to the extracted folder , and build it.

xxxxx/Downloads/scikit-learn-0.15.2$ python3.4 setup.py install --user

 








**if you get the error


 x86_64-linux-gnu-gcc: error: unrecognized command line option ‘-fstack-protector-strong’

 (Solution A:)
edit the below file by sudo

/usr/lib/python3.4/plat-x86_64-linux-gnu/_sysconfigdata_nd.py 

=>this file is belone libpython3.4-minimal package, if you don't have it,
install that.



and replace "all"  -fstack-protector-strong  to -fstack-protector

and saved.


and use the build command above again....

$  python3.4 setup.py install --user





 (Solution B:)
upgrade gcc to 4.9 (I don't want to do that.....)

[IDE] Sublime text 3 , editing config for myself. Tabs to spaces, trim white space when save, draw white sapce for notice.

1. go to  submits -> preferences -> settings - user

copy and pasted below config.


{
    "draw_white_space": "all",
    "font_size": 14,
    "translate_tabs_to_spaces": true,
    "trim_trailing_white_space_on_save": true,
    "auto_match_enabled": false,
    "use_tab_stops": false
}



2.  TrailingSpaces-master package.
Optional: highlight the white space with red color.
 (But in fact, space will be trim when you save file as above setting.)

Go to github to download the zip file for whole project on the right buttom

https://github.com/SublimeText/TrailingSpaces

then extract that and you can see the folder  TrailingSpaces-master.

Go Submits =>Preferences => Browser Packages

Then move the folder TrailingSpaces-master in to the folder which Submits jumped out.

Which means you should pasted TrailingSpaces-master folder in to

sublimes-text-3/Packages/

and place with the /sublimes-text-3/Packages/User folder together.


It will seem like....


xxxx:~/.config/sublime-text-3/Packages$ pwd
/home/xxxx/.config/sublime-text-3/Packages

xxxx:~/.config/sublime-text-3/Packages$ ll
total 24
drwx------  6 xxxx xxxx 4096  3月 22 14:43 ./
drwx------  7 xxxx xxxx 4096  3月 15 17:29 ../
drwxrwxr-x 11 xxxx xxxx 4096  3月 22 14:43 bz2/
drwxr-xr-x  7 xxxx xxxx 4096  3月 15 17:32 ssl-linux/
drwxrwxr-x  3 xxxx xxxx 4096 10月 27 22:54 TrailingSpaces-master/
drwx------  3 xxxx xxxx 4096  3月 29 09:52 User/



You are good to go.

[data analysis] 一些基本想法

統計與分析要有互相比較才有意義

比較越多 分析越正確, 對於觀察研究法尤其是如此

觀察研究法：先選定自己要當哪一群人

常常因果互想 正反推論

要知道觀察法的侷限性

[bash] check system CPU busy top 10, check sys memory status, monitor CPU usage every 2 sec

[check system CPU busy top 10]
list entire table
$ ps aux | sort -rk 3,3 | head -n 10


[Monitor the CPU usage every 2 sec]
watch "ps aux | sort -rk 3,3 | head -n 6"


[check sys memory status]

$  cat /proc/meminfo
MemTotal:        3918656 kB
MemFree:         1422332 kB
Buffers:           36868 kB
Cached:           929076 kB
SwapCached:       107124 kB
Active:          1166772 kB
Inactive:        1077372 kB






or you can check out ...

grep MemTotal /proc/meminfo | awk '{print $2}' 

[python] python 3.x read excel xls file in list.

office doc
https://secure.simplistix.co.uk/svn/xlrd/trunk/xlrd/doc/xlrd.html?p=4966

import xlrd

def open_file(path):

    """

    Open and read an Excel file

    """

    book = xlrd.open_workbook(path)

    # print number of sheets

    print book.nsheets

    # print sheet names

    print book.sheet_names()

    # get the first worksheet

    first_sheet = book.sheet_by_index(0)

    # read a row

    print first_sheet.row_values(0)

    # get how many rows in this sheet

    print first_sheet.nrows

    # read a cell

    cell = first_sheet.cell(0,0)

    print cell

    print cell.value

    # read a row slice

    print first_sheet.row_slice(rowx=0,

                                start_colx=0,

                                end_colx=2)

   return

[python] python 3.4 install xlrd to handle excel .xls file.

$ sudo gedit /etc/apt/sources.list

add below 

deb http://us.archive.ubuntu.com/ubuntu vivid main universe 

save

then

$ sudo apt-get update

$sudo apt-get install python3-xlrd 

----------------------------------------------------------------------

check the doc 

https://secure.simplistix.co.uk/svn/xlrd/trunk/xlrd/doc/xlrd.html?p=4966

[python] python 3.x install pandas under Ubuntu 14.04

$ sudo apt-get install python3-pandas


detail: http://pandas.pydata.org/pandas-docs/stable/install.html

Installing using your Linux distribution’s package manager.

The commands in this table will install pandas for Python 2 from your distribution. To install pandas for Python 3 you may need to use the package python3-pandas.

Distribution	Status	Download / Repository Link	Install method
Debian	stable	official Debian repository	sudo apt-get install python-pandas
Debian & Ubuntu	unstable (latest packages)	NeuroDebian	sudo apt-get install python-pandas
Ubuntu	stable	official Ubuntu repository	sudo apt-get install python-pandas
Ubuntu	unstable (daily builds)	PythonXY PPA; activate by: sudo add-apt-repository ppa:pythonxy/pythonxy-devel && sudo apt-get update	sudo apt-get install python-pandas
OpenSuse & Fedora	stable	OpenSuse Repository	zypper in  python-pandas

[python] 計算日期 min , Max 和求 日期差距（含假日）, check the date is what day of the week (or weekend)

from datetime import *

    # create a date( ) list
    datetime_list=[date(2016,6,30),date(2016,6,31).............]


    print(min(datetime_list),max(datetime_list))

    delta_days=max(datetime_list)-min(datetime_list)
    print(delta_days.days)

    delta_days=date(2016,3,28)-date(2016,3,21)
    print(delta_days.days)


#------------- check the date is what day of the week-----------------------------------------
    dt=date(2016,3,20)       
    if dt.isoweekday()==6 or dt.isoweekday()==7:
                print('do someting')
              



detail:   https://docs.python.org/2/library/datetime.html

[IDE] Sublime text 3 Jump to definition and back with ctrl+ mouse left click. Ubuntu/Mac/Windows

Ubuntu: 
create "Default (Linux).sublime-mousemap"
 in ~/.config/sublime-text-3/Packages/User

Mac:
create "Default (OSX).sublime-mousemap" in ~/Library/Application Support/Sublime Text 3/Packages/User


Windows
create "Default (Windows).sublime-mousemap" in %appdata%\Sublime Text 3\Packages\User



pasted below into the file you just created.

[
 {
  "button": "button1",
  "count": 1,
  "modifiers": ["ctrl"],
  "press_command": "drag_select",
  "command": "goto_definition"
 },
 {
  "button": "button2",
  "count": 1,
  "modifiers": ["ctrl"],
  "press_command": "jump_back",
  "command": "jump_back"
 }
]

 save. restart sublime text3 

Then you can press ctrl then click a function to jump to definition,
and press right click will jump back.

[python] python 用 matplotlib 畫出散點圖 scatter 筆記, 標記上點的text

matplotlib  提供能畫的種類

http://matplotlib.org/gallery.html

source code example included.


============================
import matplotlib.pyplot as plt
import numpy as np

#find max under list of list, sublist[x] means the target column
def max_value_lol(inputlist,target_column):
    return max([sublist[target_column] for sublist in inputlist])

#畫出上下兩個圖根據不同data欄位, data is list of list , head_title is list for title
def data_Visualization(head_title,data):
    #print(max_value((data)))
    max_submit_times=max_value_lol(data,1)
    point_scale=6
    #print(len(data))

    fig=plt.figure(figsize=(6,4))
    ax=fig.add_subplot(211)
    N = len(data)
    for item in data :
        x = float(item[6])
        y = float(item[7])
        colors = np.random.rand(N)
        area = np.pi * (15 * (x+y)/2)**2
        #add wording besides the points
        ax.text(x, y, item[0])
        #area = np.pi * (15 * (item[1]/max_submit_times)*point_scale)**2  # 0 to 15 point radiuses
        plt.scatter(x, y, s=area, c=colors, alpha=0.5)

    plt.xlabel(head_title[6])
    plt.ylabel(head_title[7])
    plt.xlim(-0.2,float (max_value_lol(data,6))+0.2)
    plt.ylim(-0.2,float (max_value_lol(data,7))+0.2)
    #plt.legend()




    #second pic
    ax=fig.add_subplot(212)
    for item in data :
        x = float(item[8])
        y = float(item[9])
        colors = np.random.rand(N)
        area = np.pi * (15 * (float(item[8])+float(item[9]))/15)**2
        #area = np.pi * (15 * (item[1]/max_submit_times)*point_scale)**2  # 0 to 15 point radiuses
        plt.scatter(x, y, s=area, c=colors, alpha=0.5)


    plt.xlabel(head_title[8])
    plt.ylabel(head_title[9])
    plt.xlim(-0.2,float (max_value_lol(data,8))+1)
    plt.ylim(-0.2,float (max_value_lol(data,9))+5)

    plt.show()
    return


=====================

[python][data_Visualization] python 3.x install matplotlib under ubuntu 14.04

check matplotlib still here.

$ apt-cache search python3-matplotlib

If you find it like its available then you can install it from

$ sudo apt-get install python3-matplotlib

good to go

---------------

Check what you like.

http://matplotlib.org/gallery.html

[python] python 3.x 抽象型別介紹(2) list MutableSequence Sequence 方法比較

print(dir(Sized))
['__abstractmethods__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_abc_cache', '_abc_negative_cache', '_abc_negative_cache_version', '_abc_registry']
[Finished in 0.0s]


print(dir(Container))

['__abstractmethods__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_abc_cache', '_abc_negative_cache', '_abc_negative_cache_version', '_abc_registry']
[Finished in 0.0s]


print(dir(Iterable))
['__abstractmethods__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_abc_cache', '_abc_negative_cache', '_abc_negative_cache_version', '_abc_registry']
[Finished in 0.0s]

print(dir(Sequence))
['__abstractmethods__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_abc_cache', '_abc_negative_cache', '_abc_negative_cache_version', '_abc_registry', 'count', 'index']
[Finished in 0.0s]

print(dir(MutableSequence))
['__abstractmethods__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__setattr__', '__setitem__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_abc_cache', '_abc_negative_cache', '_abc_negative_cache_version', '_abc_registry', 'append', 'clear', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse']
[Finished in 0.0s]


li=[1,2]
print(dir(li))

['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']
[Finished in 0.0s]

應該列成excel 來橫向比較比較好看

[python] python 3.4 抽象型別介紹, 實作範例

Part1: 

int 是一個type (型別), list 也是一個型別(type), 而各自不同的type 有自己能使用的method, 方法就如同前幾篇印出來看的結果.

現在, Sequence 是一個型別(type), 當然自己也有自己能用的方法.

但是, list, int 或 str... 等等這種型別, 可以自己產生物件,
ex: a=[1,2,3] => 產生了一個list 物件
ex: t=(1,2,3) =>產生了tuple 物件

而 Sequence 無法產生一個實體物件 => 因此只能算是一個抽象的型別, 概念性的, 不能真的拿來用的型別.

那你會問說, 不能用的Sequence 是要作甚呢？他就像一張賽車的設計藍圖
在Sequence的藍圖中, 畫出了賽車車輪的位置和大小, 畫出了座位的樣子等等等
但是並沒有實際的寫上細節, 沒有真正要用哪些廠牌的資訊, 因為這些細節和實際
製作的工法, 就留給那到藍圖的下游廠商去作就好, 所以換言之, 下游廠商如果接了這個
案子, 就要照著Sequence 的設計藍圖製造, 但是細節由自己決定, 可是不能有少, 否則就
沒有照這個設計圖走了.




所以, 那最源頭的抽象型別是誰呢？我們可以在 Collections Abstract Base Classes,

（https://docs.python.org/3/library/collections.abc.html）


抽象型別的集合中, 看到所有python 定義的抽象型別列表, 如下 （放在collections.abc)

ABC	Inherits from	Abstract Methods	Mixin Methods
Container		__contains__
Hashable		__hash__
Iterable		__iter__
Iterator	Iterable	__next__	__iter__
Generator	Iterator	send, throw	close, __iter__, __next__
Sized		__len__
Callable		__call__
Sequence	Sized, Iterable, Container	__getitem__, __len__	__contains__, __iter__, __reversed__, index, and count
MutableSequence	Sequence	__getitem__, __setitem__, __delitem__, __len__, insert	Inherited Sequence methods and append, reverse, extend, pop, remove, and __iadd__
Set	Sized, Iterable, Container	__contains__, __iter__, __len__	__le__, __lt__, __eq__, __ne__, __gt__, __ge__, __and__, __or__, __sub__, __xor__, and isdisjoint
MutableSet	Set	__contains__, __iter__, __len__, add, discard	Inherited Set methods and clear, pop, remove, __ior__, __iand__, __ixor__, and __isub__
Mapping	Sized, Iterable, Container	__getitem__, __iter__, __len__	__contains__, keys, items, values, get, __eq__, and __ne__
MutableMapping	Mapping	__getitem__, __setitem__, __delitem__, __iter__, __len__	Inherited Mapping methods and pop, popitem, clear, update, and setdefault
MappingView	Sized		__len__
ItemsView	MappingView, Set		__contains__, __iter__
KeysView	MappingView, Set		__contains__, __iter__
ValuesView	MappingView		__contains__, __iter__
Awaitable		__await__
Coroutine	Awaitable	send, throw	close
AsyncIterable		__aiter__
AsyncIterator	AsyncIterable	__anext__	__aiter__

 舉例來看, 在ABC這欄, 有個Sized, 也就是說, python 想營造出一個抽象的東西,
希望能夠處裡有關大小的事情, 所以我的藍圖裡想要有這個概念,
而他右邊那攔 inherits from 是沒有填, 代表Sized 這個概念是原創, 他是第一個頭,
在右邊那攔有個abstracts methods, 他填的是 __len__, 就是說, 希望後人在處裡
有關Sized 的概念時, 請將實作的function 做在__len__裡面, 所以之後, 只要想要找有關
大小的訊息時, 就去你的__len__找, 才不會大家function 名稱不一樣, 搞的亂七八糟的.

另外一個 ABC攔中的 Container 也是類似的, 就是他有個概念是, 我想要找出甚東西
包含在其中, 希望後人們能提供這個方法, 因為還蠻常用的,而你們後人要實作的話
請作在__contains__ 裡面.

那現在看Sequencs , 他繼承了 Sized, Container, Iterable, 這三個人, 所以一定會有
這三個人當初要求要作的方法,但他自己又加入了__getitem__ __len__,這兩個抽象方法

意思是說, 我序列這種類別, 將會提供sized, container, iterable, 和 getitem, len 的方法
如果你要繼承我, 之後你得做出這些東西, 而你看到 __len__不是Sized 裡面就有嗎？
怎又出現在Sequences 的 抽象方法中呢？那是因為 他仍然沒有實作, 繼續擺著, 希望後面的人能作, 而最右邊 Mixin Method, 告訴你, 我還加進了這些方法(已經做好你可以用).

而MutableSequence , 又繼承了Sequence 但列出了一些抽象型別, 等著給別人實作
__getitem__, __setitem__, __delitem__, ...等, 而可用的方法除了繼承了 Sequence
已經有的, 又加入了 append, reversed ,....等等等

這個MutableSequence 抽象型別, 也就是常聽到的[可變序列], 而list , str 都在他的下面,
所以 也就是說, list 得實作這一連串的方法, 加上list 自己想提供的方法, 而 tuple ,
因為是設計來用為不可改變的物件, 當然就不會去繼承 MutableSequence了, 但他仍然
是個序列相關的物件, 所以可見得, tuple 一定是在sequence 之下, 但實際上,
中間還有一層tuple 物件, 而 真正我們在用的實體tuple 是tuple 物件的實作.


you can use it to print the abstract class tree.

def classtree(cls):
       print(cls.__bases__)
       c = list(cls.__bases__)
       for base in c:
            c.extend(classtree(base))
       return c

classtree(MutableSequence)

output:

(<class 'collections.abc.Sequence'>,)
(<class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>)
(<class 'object'>,) #最上層了
..... #not import so I do not pasted it.


ps. 可以利用 isinstance 看是不是誰的實作, 還有issubclass 看是不是誰的子類別

li=[1,2]
t=(3,4)
print ("---check instance---")
print(isinstance(li,list), isinstance(t,tuple))
print(isinstance(li,Sequence), isinstance(t,Sequence))
print(isinstance(li,MutableSequence), isinstance(t,MutableSequence))

print ("---check list and class level---")
print(type(li),type(t))
print(issubclass(list,MutableSequence),issubclass(MutableSequence,list))
print(issubclass(MutableSequence,Sequence),issubclass(Sequence,MutableSequence))
print(issubclass(Sequence,Sized))



print ("---check tuple and class level---")
print(type(li),type(t))
print(issubclass(tuple,MutableSequence),issubclass(MutableSequence,tuple))
print(issubclass(tuple,Sequence),issubclass(Sequence,tuple))
print(issubclass(Sequence,Sized))

output:
---check instance---
True True
True True
True False
---check list and class level---
<class 'list'> <class 'tuple'>
True False
True False
True
---check tuple and class level---
<class 'list'> <class 'tuple'>
False False
True False
True
[Finished in 0.0s]



Part2 :

     假設你要實作一個Set 的實體instance, 就只要實作 __contains__, __iter__,__len__

（如表格所示）, 而其他Mixin Method 都是python 已經幫我們作好的, 直接可以用了.

所以下面的code, 要實作這三個 (init 本來自己就要), 而也看到, and 直接可以用了.

from collections.abc import *

li=[9,1,2]

class ListBasedSet(Set):
     ''' Alternate set implementation favoring space over speed
         and not requiring the set elements to be hashable. '''
     def __init__(self, iterable):
         self.elements = lst = []
         for value in iterable:
             if value not in lst:
                 lst.append(value)
     def __iter__(self):
         return iter(self.elements)
     def __contains__(self, value):
         return value in self.elements
     def __len__(self):
         return len(self.elements) ＃你在後面+ 100 , s1 印出來值也會變, 可見得真的是跑你作的這邊

s1 = len(ListBasedSet(li)) #拿你實作的來用用看
print(s1)

for x in ListBasedSet(li):  
    print(x)

 ＃你發現 & 不用實作, 直接可以用,因為 __and__在Mixin Method 裡面已經註明他做好了

for x in (ListBasedSet('abcdef') & ListBasedSet('defwxyz')): 
    print(x)

output:

3
9
1
2

d
e
f
[Finished in 0.1s]