# README ## Static Sites * Built by MkDocs & CF Pages: or * Built by gitbook: view on [github](https://github.com/fzinfz/book) or [vscode](https://github1s.com/fzinfz/book) if unexpected html rendering ## Home Visit home for more: # DevOps * [UI](#ui) * [Grafana - TS + Go](#grafana---ts--go) * [Monitoring](#monitoring) * [Prometheus - Go + TS](#prometheus---go--ts) * [InfluxDB Telegraf](#influxdb-telegraf) * [Grafana Promtail](#grafana-promtail) * [Zabbix - C/PHP/JAVA](#zabbix---cphpjava) * [server - Docker](#server---docker) * [agent](#agent) * [Nagios - C](#nagios---c) * [Docker](#docker) * [Elastic](#elastic) * [Beats - Go](#beats---go) * [Cacti - PHP](#cacti---php) * [alert](#alert) * [TICK stack](#tick-stack) * [Pandora FMS - PHP/Perl](#pandora-fms---phpperl) * [open-falcon - Go + Python Flask](#open-falcon---go--python-flask) * [Munin - Perl/Shell](#munin---perlshell) * [netdata - C/Python/JS/Shell](#netdata---cpythonjsshell) * [Management](#management) * [Fabric - Python library](#fabric---python-library) * [invoke - Python library](#invoke---python-library) * [Terraform](#terraform) * [Ansible - Python](#ansible---python) * [Puppet - Ruby](#puppet---ruby) * [Chef - Ruby](#chef---ruby) * [SaltStack - Python](#saltstack---python) * [CI](#ci) * [Jenkins - JAVA](#jenkins---java) * [Travis - Ruby/JS](#travis---rubyjs) * [SNMP](#snmp) * [More](#more) ## UI ### Grafana - TS + Go * Search dashboards & copy ID to /dashboard/import: * no login: ``` [auth.anonymous] enabled = true org_name = Main Org. org_role = Viewer hide_version = true # default: false ``` ## Monitoring ### Prometheus - Go + TS Grafana: | {url}?var-node= ### InfluxDB Telegraf ### Grafana Promtail agent : local logs -> [loki](/db/loki) : ### Zabbix - C/PHP/JAVA #### server - Docker ``` docker run --name zabbix-appliance -t \ -p 10051:10051 \ -p 8083:80 \ -d zabbix/zabbix-appliance:latest # Default login: Admin/zabbix ``` #### agent ``` Active : zabbix_agentd: active checks -> zabbix_server: trapper :10051 ServerActive= Passive: zabbix_server: poller -> zabbix_agentd: listener :10050\ Server= # Windows agent, run under admininstrator cmd zabbix_agentd.exe --config zabbix_agentd.win.conf --install # Debian/Ubuntu apt install zabbix-agent service zabbix-agent start ``` ### Nagios - C #### Docker ``` docker run --name nagios4 --rm -it -p 0.0.0.0:8082:80 jasonrivers/nagios:latest docker cp nagios4:/opt/nagios/etc ./nagios/etc docker cp nagios4:/opt/nagios/var ./nagios/var docker cp nagios4:/opt/nagiosgraph/etc ./nagios/graph_etc docker cp nagios4:/opt/nagiosgraph/var ./nagios/graph_var docker run --name nagios4 \ -d --restart unless-stopped \ -v $PWD/nagios/etc/:/opt/nagios/etc/ \ -v $PWD/nagios/var:/opt/nagios/var/ \ -v $PWD/nagios/graph_etc:/opt/nagiosgraph/etc \ -v $PWD/nagios/graph_var:/opt/nagiosgraph/var \ -v $PWD/nagios/custom-plugins:/opt/Custom-Nagios-Plugins \ -p 8082:80 jasonrivers/nagios:latest docker exec -it nagios4 htpasswd /opt/nagios/etc/htpasswd.users nagiosadmin docker exec -it nagios4 cat /opt/nagios/etc/objects/contacts.cfg docker exec -it nagios4 grep ^cfg_ /opt/nagios/etc/nagios.cfg docker restart nagios4 && docker logs nagios4 # nrpe NAGIOS_SERVER=1.2.3.4 docker run -d --restart unless-stopped \ -v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys:ro \ -v /var/lib/docker/:/var/lib/docker:ro \ --privileged --net=host --ipc=host --pid=host \ -e NAGIOS_SERVER="$NAGIOS_SERVER" \ --name nagios_nrpe \ mikenowak/nrpe ``` ### Elastic #### Beats - Go ``` Filebeat Log Files Beats Metricbeat Metrics Beats Packetbeat Network Data Beats Winlogbeat Windows Event Logs Beats Auditbeat Audit Data Beats Heartbeat Uptime Monitoring ``` ### Cacti - PHP ``` docker pull smcline06/cacti:latest ``` #### alert ### TICK stack | Component | Role | | ---------- | -------------- | | Telegraf | Data collector | | InfluxDB | Stores data | | Chronograf | Visualizer | | Kapacitor | Alerter | ### Pandora FMS - PHP/Perl ``` # Auto docker curl -sSL http://pandorafms.org/getpandora | sh # Auto, or manually below # Manually docker run \ --name pandora-mysql \ -e MYSQL_ROOT_PASSWORD=AVeryStrongRootPassword \ -e MYSQL_DATABASE=pandora \ -e MYSQL_USER=pandora \ -e MYSQL_PASSWORD=pandora -d pandorafms/pandorafms-mysql:6 docker run -p 41121:41121 \ --link pandora-mysql:mysql \ -d pandorafms/pandorafms-server:6 docker run \ -p 80:80 -p 8022:8022 -p 8023:8023 \ --link pandora-mysql:mysql \ -d pandorafms/pandorafms-console:6 apt install -y pandorafms-agent ``` ### open-falcon - Go + Python Flask v0.3: May 30, 2019 ### Munin - Perl/Shell networked resource monitoring tool\ ### netdata - C/Python/JS/Shell (with screenshots)\ ``` bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh) ``` ## Management ### Fabric - Python library Fabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return. Fabric (1.x and earlier) was a hybrid project implementing two feature sets: task execution (organization of task functions, execution of them via CLI, and local shell commands) and high level SSH actions (organization of servers/hosts, remote shell commands, and file transfer). ### invoke - Python library When planning Fabric 2.x, having the “local” feature set as a standalone library made sense, and it seemed plausible to design the SSH component as a separate layer above. Thus, Invoke was created to focus exclusively on local and abstract concerns, leaving Fabric 2.x concerned only with servers and network commands. ### Terraform * * Self Managed, always free: ### Ansible - Python ### Puppet - Ruby ### Chef - Ruby ### SaltStack - Python Agentless: ## CI ### Jenkins - JAVA ![](https://jenkins.io/images/blueocean/blueocean-successful-pipeline.png) ``` docker run -d -p 8089:8080 -p 50000:50000 jenkins/jenkins:lts docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword docker run jenkins/jnlp-slave -url http://jenkins-server:port ``` ``` Open a browser on the slave machine and go to the Jenkins master server url (http://yourjenkinsmaster:8080). Go to Manage Jenkins > Manage Nodes, Click on the newly created slave machine. You will need to login as someone that has the "Connect" Slave permission if you have configured global security. Click on the Launch button to launch agent from browser on slave. ``` run on all nodes: elastic-axis ### Travis - Ruby/JS ## SNMP v1: Authentication of clients is performed only by a "community string", in effect a type of password, which is transmitted in cleartext.\ v2c comprises SNMPv2 without the controversial new SNMP v2 security model, using instead the simple community-based security scheme of SNMPv1. incompatible with SNMPv1 in two key areas: message formats and protocol operations.\ v2u: greater security than SNMPv1, but without incurring the high complexity of SNMPv2.\ v3 primarily added security and remote configuration enhancements to SNMP. the agent connects to the server on port 162\ port 161 on the agent side is used for queries ![](https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/SNMP_communication_principles_diagram.PNG/1000px-SNMP_communication_principles_diagram.PNG) ## More # Grafana\_Cloud * [Price](#price) * [Grafana](#grafana) * [Loki Logs](#loki-logs) * [Tempo Traces](#tempo-traces) * [k6 Testing](#k6-testing) * [Pyroscope Profiles](#pyroscope-profiles) ## Price * Free: Grafana + Loki + Tempo + K6 + Pyroscope | 10k c * Pro: Prometheus + Graphite + Alerts ## Grafana * Free 3 Monthly Active Users Grafana Agent Role: `grafana.grafana.grafana_agent` * ## Loki Logs * Free 50 GB Logs ## Tempo Traces Free 50 GB Traces ## k6 Testing * Free 500 k6 Virtual User Hours * ## Pyroscope Profiles * Free 50 GB Profiles ![](https://grafana.com/img/pyroscope_agent_server_diag.png) # OS\_depoly * [netboot.xyz](#netbootxyz) * [iVentoy](#iventoy) ## netboot.xyz ## iVentoy * Free vs Paid: * OS: # ansible * [Install](#install) * [CLI](#cli) * [UI](#ui) ## Install ### CLI ``` uv tool install ansible + ansible==13.6.0 + ansible-core==2.20.5 ``` ### UI * # OpenWrt * [Install on X86](#install-on-x86) * [run as Container](#run-as-container) * [macvlan - access host](#macvlan---access-host) * [ipvlan](#ipvlan) * [run as VM](#run-as-vm) * [QEMU NIC](#qemu-nic) * [Docker](#docker) * [network](#network) * [DSA](#dsa) * [/etc/config/](#etcconfig) * [DHCP/DNS](#dhcpdns) * [QoS](#qos) * [SQM](#sqm) * [nftables](#nftables) * [Tailscale](#tailscale) * [Mesh](#mesh) * [bat-adv](#bat-adv) * [Mode 802.11s](#mode-80211s) * [Mode AP - 802.11r](#mode-ap---80211r) * [Switch Chip](#switch-chip) * [Bridged AP Setup](#bridged-ap-setup) * [Controller - OpenWISP](#controller---openwisp) * [Compile](#compile) * [Version](#version) * [Wireshark](#wireshark) ## Install on X86 ``` dd if=openwrt-21.02.0-x86-64-generic-ext4-combined.img bs=1M of=/dev/sdX opkg update opkg install lsblk parted losetup resize2fs echo fix | parted -l ---pretend-input-tty parted -s /dev/sda resizepart 2 100% losetup /dev/loop1 /dev/sda2 resize2fs -f /dev/loop1 ``` ## run as Container * * ``` ip link set vlan.10 promisc on docker network create -d macvlan --subnet=10.0.0.0/8 --gateway=10.0.0.1 -o parent=vlan.10 macnet docker network ls && docker network inspect macnet docker run --restart unless-stopped --name openwrt -d --network macnet --privileged sulinggg/openwrt:x86_64 /sbin/init # root/password docker exec -it openwrt /bin/sh # vim /etc/config/network // edit ip/gw & restart config interface 'lan' option type 'bridge' option ifname 'eth0' option proto 'static' option ipaddr '10.19.0.3' option netmask '255.0.0.0' option gateway '10.0.0.1' option broadcast '10.255.255.255' option dns '10.0.0.1' docker network inspect macnet ``` ### macvlan - access host ``` docker network create -d macvlan -o parent=eno1 \ --subnet 192.168.1.0/24 \ --gateway 192.168.1.1 \ --ip-range 192.168.1.192/27 \ --aux-address 'host=192.168.1.223' \ mynet ip link add macnet-shim link vlan.10 type macvlan mode bridge ip addr add 10.19.0.1/8 dev macnet-shim ip link set macnet-shim up ip route add 10.0.0.1/8 dev macnet-shim ip link show macnet-shim || ip link delete macnet-shim ``` macvlan/ipvlan: ### ipvlan ``` docker network create -d ipvlan \ --subnet=10.0.0.0/8 \ --gateway=10.0.0.1 \ --ip-range=10.19.1.0/24 \ -o ipvlan_mode=l2 \ -o parent=vlan.10 ipvlan10_NotTested ``` ## run as VM ### QEMU NIC ``` use e1000; rtl8139 not detected by default. eth0 -> LAN, eth1 -> WAN, usually. ``` ## Docker ``` service dockerd stop vi /etc/config/dockerd # data_root vi /etc/docker/daemon.json # data-root reboot docker pull hello-world ``` ## network ### DSA replace swconfig Multiple networks (using VLANs): ### /etc/config/ ``` config interface 'wan' option ifname 'eth0' option proto 'dhcp' option 'defaultroute' '1' # if multi WAN config interface 'lan' option type 'bridge' option ifname 'eth1 eth2' option proto 'static' option ipaddr '192.168.99.1' option netmask '255.255.255.0' option ip6assign '60' /etc/init.d/network restart # Soft network reload service network reload # Hard network restart service network restart ``` H/W Router: wireless interfaces may be added to lan automatically via LUCI, create new for other bridges. ## DHCP/DNS /etc/config/dhcp ``` config host list mac 'xx:xx:xx:xx:xx:xx' option ip '192.168.6.16' config domain option name 'homeassistant.local' option ip '192.168.6.16' ``` ## QoS ### SQM * Interface name to your internet (WAN) link * Link Layer Adaptation: ### nftables ## Tailscale * * ## Mesh ``` iw list | grep -E "phy|mesh" # check if supported hardware ``` ### bat-adv ``` opkg remove wpad-basic- opkg install batctl-full kmod-batman-adv wpad-mesh-wolfssl ``` ### Mode 802.11s ### Mode AP - 802.11r [./wireless.md#80211kvr](/openwrt/wireless#80211kvr) ## Switch Chip ``` swconfig list swconfig dev switch0 show VLAN 1: vid: 1 ports: 0 1 6 # 6 = untagged CPU VLAN 10: # luci： `/network/vlan` vid: 10 ports: 2 3 6t # tag CPU => create `eth0.X`(eth0=switch0) under `/network/iface_add` ``` * Each port `untagged` to exactly one VLAN ID ### Bridged AP Setup | Web URI | Task | | ------------------------------ | ------------------------------------ | | /luci/admin/network/vlan | + vlan : all ports tagged | | /luci/admin/network/iface\_add | test new vlan IP ; lan : remove dhcp | ## Controller - OpenWISP * Install: * Features: * Config: ## Compile ``` git clone --single-branch --branch main --depth 1 https://github.com/openwrt/openwrt.git /data/github/openwrt git clone --single-branch --branch 22.03 --depth 1 https://github.com/Lienol/openwrt.git /data/github/openwrt-Lienol-22.03 git clone --single-branch --branch master --depth 1 https://github.com/coolsnowwolf/lede /data/github/openwrt-lede ``` ``` docker run -itd \ --name openwrt-build \ -v /data/github/openwrt:/home/user/openwrt \ p3terx/openwrt-build-env n=openwrt-build-lede docker exec $n sudo chown -hR user:user . && docker exec -it $n bash # tmux cd ~/openwrt && ls -la # make clean # rm /bin /build_dir ./scripts/feeds update -a ; ./scripts/feeds install -a make menuconfig # make targetclean make download -j8 V=s && make V=s -j$(($(nproc) - 1)) ls /data/github/openwrt*/bin/targets/mediatek/mt7622/*.bin -lh # host ``` LuCI ---> Applications ---> luci-app-mtwifi #闭源Wi-Fi驱动 + kmod-mt76... Extra packages ---> ipv6helper ### Version ``` CONFIG_VERSIONOPT=y CONFIG_IMAGEOPT=y CONFIG_VERSION_DIST="##.##-SNAPSHOT" CONFIG_VERSION_NUMBER="OpenWrt" ``` ## Wireshark * UI: click left icon of text `SSH remote capture` (no password save) * CMD: # DHCP\_DNS * [Docs](#docs) * [Static](#static) * [Luci](#luci) * [uci](#uci) * [Lease Time](#lease-time) * [per client](#per-client) * [remove active](#remove-active) ## Docs ## Static /etc/config/dhcp ### Luci * all conf: * dynamic -> static : ### uci ``` uci show | grep dhcp | grep host uci add dhcp host # will create `dhcp.@host[INDEX]=host` uci set dhcp.@host[-1].ITEM # -1 : last | -x: xth from last uci delete dhcp.@host[-1] uci commit dhcp # refresh Luci service dnsmasq restart # reconnect client ``` ## Lease Time ``` cat /tmp/dhcp.leases uci show | grep dhcp | grep lease => dhcp.lan.leasetime='12h' uci set dhcp.lan.leasetime='10m' uci commit dhcp; service odhcpd restart ; service dnsmasq restart ``` ### per client luci: ### remove active ``` vi /tmp/dhcp.leases /etc/init.d/dnsmasq restart ``` # OpenVPN ``` VPN_PKI="/etc/easy-rsa/pki" export EASYRSA_PKI="${VPN_PKI}" export EASYRSA_CERT_EXPIRE="36500" cp -p pki/private/*.key /etc/openvpn/ cp -p pki/issued/*.crt /etc/openvpn/ cp -p pki/{ca.crt,dh.pem} /etc/openvpn/ service openvpn restart ``` # boot * [BL2/FIP](#bl2fip) * [MTD](#mtd) * [uboot](#uboot) * [cmd](#cmd) * [Breed](#breed) ## BL2/FIP * Boot Loader stage 1 (BL1) AP Trusted ROM * Boot Loader stage 2 (BL2) Trusted Boot Firmware * Firmware Image Package ## MTD calc HEX -> DEC ： 00400000 = 4MiB 06f00000 = 111MiB ``` cat /proc/mtd fw_printenv | grep mtdparts mtd write .fip FIP # kmod-mtd-rw dd if=/tmp/uboot.fip of=/mtdX ``` ## uboot ## cmd ``` ubinfo -a smeminfo ``` ## Breed breed -> openwrt initramfs -> /cgi-bin/luci/admin/system/flashops/sysupgrade # captive\_portal * [CoovaChilli](#coovachilli) * [uspot](#uspot) * [NoDogSplash (NDS)](#nodogsplash-nds) ## CoovaChilli * * Config: ``` # /etc/config/chilli # HS_MACALLOW="..." # List of MAC addresses to authenticate (comma seperated) /etc/init.d/chilli status ``` ## uspot drop-in replacement for CoovaChilli / nftables , RFC8908 Captive Portal API * * credentials : a simple username/password authentication * click-to-continue * radius * uam : remote RADIUS / MAC-based authentication bypass ## NoDogSplash (NDS) a simple way to provide restricted access to the Internet by showing a splash page to the user before Internet access is granted. * * Screenshot: # hw * [Chips](#chips) * [MediaTek](#mediatek) * [CN](#cn) * [OpenWrt One](#openwrt-one) * [GLiNet](#glinet) * [Routers](#routers) * [VLAN](#vlan) * [Multi WAN](#multi-wan) ## Chips ### MediaTek * Ethernet switch * MT7531AE: 2.5Gbps / SerDes interface * MT7531BE: CPU port / RGMII (Reduced Gigabit Media Independent Interface) MT7981B ds: ## CN * Xiaomi / Redmi: * gl-inet: | openwrt.org | SoC | CPU MHz | Flash MB | RAM MB | Wireless | Switch | | ---------------------------------------------------------------------- | -------- | ------- | -------- | ------ | ----------------- | -------- | | [CT3003](https://openwrt.org/toh/hwdata/cetron/cetron_ct3003) | MT7981B | 2c1.3 | 128 | 256 | MT7981 | MT7531AE | | [RM AX6S](https://openwrt.org/toh/xiaomi/ax3200) | MT7622B | 2c1.35 | 128NAND | 256 | MT7622B/MT7915E | MT7531BE | | [Mi AX3000T](https://openwrt.org/inbox/toh/xiaomi/ax3000t) | MT7981BA | 2c1.3 | 128NAND | 256 | MT7981BA/MT7976CN | MT7531AE | | [GL-MT3000](https://openwrt.org/toh/gl.inet/gl-mt3000) | MT7981BA | 2c1.3 | 256 | 512 | MT7981BA | MT7981BA | | [RM AX6](https://openwrt.org/inbox/toh/xiaomi/xiaomi_redmi_ax6_ax3000) | IPQ8071A | 4c1.4 | 128 MiB | 512 | QCN5024/QCN5054 | QCA8075 | | [GL-MT6000](https://openwrt.org/toh/gl.inet/gl-mt6000) | MT7986A | 4c2.0 | 8G eMMC | 1024 | MT7976GN/MT7976AN | MT7531AE | ## OpenWrt One November 29, 2024: MT7981B / 2C 1.3GHz / 1G + 256MB / LAN: 2.5G + 1G : ## GLiNet ``` ls -lh /etc/oui-tertf/client.db /etc/config ``` API: Cloud Web: set firewall/lan subnet/ ; view clients * CN : * * remote web/terminal : need pub ip? : ### Routers GL-MT6000 | eth1 + ( eth0 : lan1-5 ) GL-MT3000 | eth0 2.5G + eth1 1G * swap: /#/netport make eth0 lan -> luci , don't edit /etc/config/network directly! ### VLAN br-lan: rm physical, add br-VLANs.1 * if br-lan.1 : GL UI won't show clients ### Multi WAN no custom rules on luci * wan * secondwan * wwan * tethering | USB | ``` cat /etc/config/kmwan option level # TODO cat /etc/hotplug.d/iface/99-kmwan uci -q get kmwan.global.enable ``` sensitivity: detection time interval(unit:s) # luci * [Web](#web) * [Troubleshooting](#troubleshooting) * [port](#port) * [http](#http) * [/www/](#www) * [/usr/lib/lua/luci/](#usrliblualuci) ## Web | Action | Addr | | ------------- | --------------------------------------------------------------- | | conf firewall | | | view iptables | | | view conn | | | SSH key | | /etc/config/luci * ping/traceroute/nslookup: ### Troubleshooting * * rm default gw of interface if unexpected 0.0.0.0/0 * firewall + DNS ## port ### http ``` grep listen /etc/config/uhttpd /etc/nginx/conf.d/* ``` ## /www/ * cgi-bin/luci ## /usr/lib/lua/luci/ * model/view/controller * admin # mwan3 * [Doc](#doc) * [Install](#install) * [Management](#management) * [Debug](#debug) ## Doc * A new routing table is created for each interface * A monitoring script (mwan3track) runs in the background checking each WAN interface * * ## Install * [mt7981](https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/packages/) ## Management `/cgi-bin/luci/admin/network/mwan3` * Globals * Notify: /etc/mwan3.user(.sh) * Rule -> Policy -> Member ( compare `weight` if same `metric`) -> Interface ## Debug `/cgi-bin/luci/admin/status/mwan3`: status/Diagnostics ``` ip route show # Interface ``` # radius ## platform * ## FreeRADIUS * clients.conf - * client private-network-\* { ipaddr + secret * sites-enabled/default * listen { * authorize { filter\_password # * files: mods-config/files/authorize * PAP (Password Authentication Protocol) | `pap: WARNING: Authentication will fail unless a "known good" password is available` opkg update && opkg install freeradius3-default apt install -y freeradius-utils # client tools ls -R /etc/freeradius3 /etc/init.d/radiusd stop && radiusd -X # debug radtest bob hello wrt.lan 0 testing123 # 0 = NAS-port-number = any # route * [PBR](#pbr) * [OSPF](#ospf) * [frr-ospfd](#frr-ospfd) * [bird2](#bird2) ## PBR * PBR app * split tunneling * PBR with netifd * route traffic to a specific interface * mwan3 * load balancing and failover ## OSPF ### frr-ospfd FRRouting * active fork of Quagga ### bird2 BIRD Internet Routing Daemon * lightweight # theme * [Create](#create) ## Create # wireless * [Roaming](#roaming) - [Debug](#debug) * [802.11kvr](#80211kvr) - [/etc/config/wireless](#etcconfigwireless) - [MediaTek](#mediatek) * [Client Info](#client-info) * [Controller](#controller) - [DAWN](#dawn) - [usteer](#usteer) - [fakemesh](#fakemesh) ## Roaming ### Debug * * ## 802.11kvr [/nw/README.md#roaming---80211krv](/nw#roaming---80211krv) ### /etc/config/wireless * helper: * uci: * [debug](https://forum.openwrt.org/t/any-way-to-monitor-if-802-11r-is-working/73504): [hostapd log](https://openwrt.org/docs/guide-developer/debugging#logging_hostapd_behaviour) ``` # grep -E '11(k|v|r)|^config' /etc/config/wireless config wifi-device 'radio0' config wifi-iface 'default_radio0' option ieee80211r '1' option ieee80211k '1' # option ieee80211v Removed ``` ### MediaTek ``` grep -E 'RRM|WNM|Ft' -r /etc/wireless/ # k/v/r ``` ## Client Info ``` # fping -g 192.168.8.0/24 # show IPv4 if IPv6 under `Associated Stations` # iw dev | grep -E 'Interface|ssid' # iw dev phy1-ap0 station dump Station [K60_MAC] (on phy1-ap0) inactive time: 10 ms rx bytes: 1274880 rx packets: 7828 tx bytes: 56193821 tx packets: 39844 tx retries: 8088 tx failed: 8088 rx drop misc: 6 signal: -83 [-86, -91, -86, -95] dBm signal avg: -82 [-86, -91, -86, -95] dBm tx bitrate: 432.3 MBit/s 160MHz HE-MCS 2 HE-NSS 2 HE-GI 0 HE-DCM 0 tx duration: 21709248 us rx bitrate: 432.3 MBit/s 160MHz HE-MCS 2 HE-NSS 2 HE-GI 0 HE-DCM 0 rx duration: 542587 us last ack signal:-12 dBm avg ack signal: -22 dBm airtime weight: 256 authorized: yes authenticated: yes associated: yes preamble: short WMM/WME: yes MFP: no TDLS peer: no DTIM period: 2 beacon interval:100 short preamble: yes short slot time:yes connected time: 87 seconds associated at [boottime]: 283658.981s associated at: 1700846966098 ms current time: 1700847053027 ms ``` ## Controller ### DAWN Decentralized WiFi Controller * * full wpad-\* version and not wpad-basic * * ``` sed -i 's/10.0.0.255/192.168.88.255/' /etc/config/dawn && /etc/init.d/dawn restart grep -E '_port|_ip' /etc/config/dawn && netstat -lntup | grep dawn ``` ### usteer ### fakemesh * Sync Config # ai * [Model Ranking](#model-ranking) * [Concepts](#concepts) * [Links](#links) ## Model Ranking ## Concepts * MCP (Model Context Protocol) * connecting AI applications to external systems * ## Links * Course: * GPT: # ML * [NTU Courses](#ntu-courses) * [Linear Algebra](#linear-algebra) * [Keras](#keras) * [scikit-learn](#scikit-learn) ## NTU Courses ### Linear Algebra ## Keras ## scikit-learn (TW) # agent * [Compare](#compare) * [OpenClaw](#openclaw) ## Hermes Agent ``` curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash hermes dashboard --host 0.0.0.0 --port 9111 --no-open --insecure ``` ## OpenSquilla Token-Efficient: * routes across four tiers (T0–T3) to the cheapest capable model ## Compare OpenSquilla OpenClaw Hermes Agent : | | [OpenClaw](https://github.com/openclaw/openclaw) | [NanoBot](https://github.com/HKUDS/nanobot) | [PicoClaw](https://github.com/sipeed/picoclaw) | [ZeroClaw](https://github.com/zeroclaw-labs/zeroclaw) | [NullClaw](https://github.com/nullclaw/nullclaw) | | --------------------- | ------------------------------------------------ | ------------------------------------------- | ---------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------ | | **Language** | TypeScript | Python | Go | Rust | **Zig** | | **RAM** | > 1 GB | > 100 MB | < 10 MB | < 5 MB | **\~1 MB** | | **Startup (0.8 GHz)** | > 500 s | > 30 s | < 1 s | < 10 ms | **< 8 ms** | | **Binary Size** | \~28 MB (dist) | N/A (Scripts) | \~8 MB | 3.4 MB | **678 KB** | ## OpenClaw origin not allowed (open the Control UI from the gateway host or allow it in gateway.controlUi.allowedOrigins) ``` "gateway.controlUi.allowedOrigins": [ "http://127.0.0.1:37798" // 本地 ] ``` # coding * [IDE](#ide) * [CN](#cn) * [USD](#usd) * [API](#api) * [openrouter](#openrouter) * [TTS](#tts) * [MiMo-V2-TTS](#mimo-v2-tts) * [Tools](#tools) * [Gemini CLI](#gemini-cli) ## IDE ### CN | 工具 | 出品方 | IDE | ext | free | | ---------------- | ----- | ------- | ---- | ------------ | | 通义灵码 (Qoder CN)​ | 阿里云 | +Quest | full | 2week pro | | CodeBuddy​ | 腾讯云 | +Agents | - | manual/daily | | Trae（国内版）​ | 字节跳动 | +SOLO | full | | | 文心快码 Comate | 百度 | - | full | | | CodeGeeX | 智谱 AI | - | full | | ## USD | Price | Site | 1M tokens | | ----------- | ----------------------------------------------- | ------------------------------------ | | 0/PAYG | | $0.25(text/image/video) $0.50(audio) | | 0/8/20/250 | | Antigravity | | 0/10/39 | | | | 0/8/20/200 | | | | 0/20/60/200 | | | | 0/15/30 | | | | 0/10 | | | | 0/20 | | | | 0/PAYG | | API | | 0/17 | | non-CN | | 20+ | | | ## API ### openrouter * * ## TTS ### MiMo-V2-TTS ## Tools | Feature / Tool | **Cline** | **OpenCode** | **Kilocode** | **OpenClaw** | **Blackbox AI** | | ----------------------- | --------------------------- | -------------------------- | ------------------------- | --------------------------------- | -------------------------------- | | **Primary Interface** | VS Code Extension | Terminal (CLI / TUI) | VS Code, JetBrains, CLI | OS / Terminal Background | Web, Mobile, IDE Extension | | **Philosophy** | Human-in-the-loop IDE Agent | Terminal-first local Agent | Unified Agentic Workspace | Autonomous Background Assistant | Multi-modal Rapid Generator | | **Open Source** | ✅ Yes | ✅ Yes | ✅ Yes (Core) | ✅ Yes | ❌ Proprietary (Freemium) | | **Standout Superpower** | Diff-based UI approvals | Deep Terminal integration | Cross-IDE syncing & modes | Truly autonomous background tasks | Image/Figma-to-Code & Mobile App | ### Gemini CLI * free: 60 requests/min & 1,000 requests/day # free * 200K Context Length | # tool ## mix ### tokscale ``` npx tokscale@latest ``` ## opencode ``` uv tool install opencode-monitor uv tool upgrade opencode-monitor pipx install opencode-monitor # Arch Linux, Ubuntu, macOS, etc # Real-time monitoring dashboard ocmonitor live # View sessions history ocmonitor sessions # Daily usage breakdown (add --breakdown for per-model detail) ocmonitor daily ocmonitor daily --breakdown # Deep dive into a specific model ocmonitor model claude-sonnet-4-5 ``` # apps * [Screen sharing](#screen-sharing) * [Shell](#shell) * [Terminal](#terminal) * [Windows](#windows) * [KVM switch](#kvm-switch) * [File Transfer](#file-transfer) * [Note](#note) * [NextCloud](#nextcloud) * [Sync](#sync) * [ResilioSync/BTSync](#resiliosyncbtsync) * [GPU](#gpu) * [OpenCL](#opencl) * [bench](#bench) * [net](#net) * [DSM](#dsm) * [Security](#security) * [Password Managment](#password-managment) * [Bitwarden](#bitwarden) * [Instant Notifications](#instant-notifications) * [IoT](#iot) * [Taobao](#taobao) * [Sandbox](#sandbox) ## Screen sharing * host | FreeBSD Linux macOS Windows: * client: * qt: | Type | Site | Server | Client | Management | | ---------- | ------------------------ | ------------- | ------------------- | ---------- | | OSS | | win/mac/linux | web | ViewOnly | | Commercial | | win | win/ios/android/web | Touch | ### Shell ## Terminal ### Windows C | X11/XYZModem: TypeScript | Terminus: Free version allowd in companay: Free for home: ## KVM switch Barrier was forked from Symless's Synergy 1.9 codebase. h/w: pikvm ## File Transfer ## Note ### NextCloud if behind rproxy:\ ``` 'overwritehost' => '...', 'overwriteprotocol' => 'https', ``` ## Sync ### ResilioSync/BTSync ## GPU ### OpenCL ``` phoronix-test-suite benchmark pts/opencl phoronix-test-suite test system/opencl # https://download.blender.org/demo/test/cycles_benchmark_20160228.zip ``` ## bench ### net ## DSM * UEFI q35 + USB boot + SATA data disk Ports: ## Security Password Crack: ### Password Managment #### Bitwarden lightweight: clients/extensions: ## Instant Notifications ## IoT ## Music * 网易云音乐 3.0 之后的某些版本，下载的 ncm 文件会出现不内置歌曲专辑的封面图片 ## Sandbox ??%SystemDrive%\Sandbox%USER%%SANDBOX% # web * [Tools](#tools) * [Search Engines](#search-engines) * [Linux](#linux) ## Tools | Desc | Link | | ------------------------- | ---------------------------------------------- | | Table to Markdown | | | Docker Cmd Helper | | | text split/filter/replace | | | .md <-> .rst | | | YAML / JSON / TOML | | ## Search Engines ## Linux * cron explain: * # BSD * [Desktop](#desktop) * [TCP congestion control](#tcp-congestion-control) * [relayd](#relayd) * [PF](#pf) * [ipfw](#ipfw) * [ZFS](#zfs) * [DTrace](#dtrace) * [CAM Target Layer(ctl)](#cam-target-layerctl) * [HA cluster](#ha-cluster) ## Desktop * FreeBSD since 2010, based on FreeBSD * TrueOS (formerly PC-BSD or PCBSD) ended in 2020 ## TCP congestion control ``` sysctl net.inet.tcp.cc net.inet.tcp.cc.available: newreno net.inet.tcp.cc.algorithm: newreno kldload cccubic kldload ccvegas kldload cccdg ``` ## relayd layer 3 and/or layer 7 load-balancer, application layer gateway, or transparent proxy ## PF ``` pass in all pass in from any to any pass in proto tcp from any port < 1024 to any pass in proto tcp from any to any port 25 pass in proto tcp from 10.0.0.0/8 port >= 1024 \ to ! 10.1.2.3 port != ssh pass in proto tcp from any os "OpenBSD" pass in proto tcp from route "DTAG" ``` ### ipfw ``` sudo ipfw add fwd 127.0.0.1,12345 tcp from not me to any 80 in via en1 sudo ipfw add fwd 127.0.0.1,12345 tcp from not me to any 443 in via en1 ``` ## ZFS ``` zpool create pool_name da0p3 da1p3 zpool create pool_name \ mirror da0 da1 \ mirror da2 da3 \ log mirror da4 da5 zpool add pool cache da2 da3 zpool list zpool get all pool_name zpool status pool_name zpool iostat -v pool_name 5 zpool list -v pool_name ``` ## DTrace ## CAM Target Layer(ctl) The ctl subsystem provides SCSI target devices emulation ### HA cluster # Mac * [HighDPI](/bsd/mac#highdpi) * [pip](/bsd/mac#pip) * [brew & gnu tools](/bsd/mac#brew--gnu-tools) * [Hackintosh](/bsd/mac#hackintosh) * [Build an OS X boot disk](/bsd/mac#build-an-os-x-boot-disk) * [On QEMU/KVM](/bsd/mac#on-qemukvm) ## HighDPI ``` sudo defaults write /Library/Preferences/com.apple.windowserver.plist DisplayResolutionEnabled -bool true ``` ## pip ``` wget https://bootstrap.pypa.io/3.2/get-pip.py sudo python get-pip.py ``` ## brew & gnu tools ``` /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" export PATH="$(brew --prefix coreutils)/libexec/gnubin:/usr/local/bin:$PATH" brew install findutils --with-default-names brew install gnu-indent --with-default-names brew install gnu-sed --with-default-names brew install gnutls brew install grep --with-default-names brew install gnu-tar --with-default-names brew install gawk ``` ## Hackintosh ### Build an OS X boot disk ### On QEMU/KVM # Cloud * [Domain Name lookup](#domain-name-lookup) * [Object Storage](#object-storage) * [HIDS uninstall](#hids-uninstall) * [jcloud](#jcloud) * [aliyun](#aliyun) * [qcloud](#qcloud) ## Domain Name lookup ## Object Storage China：tx COS | ali/jd OSS | hw OBS ## HIDS uninstall ### jcloud ``` wget http://hids.s-sq.jcloud.com/jcloudhids_linux64_V1.0.tar.gz tar zxvf jcloudhids_linux64_V1.0.tar.gz ./jcloudhids_linux64_V1.0.19216/uninstall.py ``` ### aliyun ``` rm -f /usr/local/share/aliyun-assist/1.0.1.259/aliyun-service ``` ### qcloud ``` /usr/local/sa/agent /usr/local/qcloud/monitor/barad/admin ``` # aws * [lambda](#lambda) ## lambda # azure * [Console](#console) * [Services free for 12 months](#services-free-for-12-months) * [storage](#storage) * [cli](#cli) * [vm](#vm) * [disk](#disk) * [network lb](#network-lb) * [inbound-nat-rule](#inbound-nat-rule) ## Console ### Services free for 12 months -- click subscription -> "Overview" ## storage | | Ultra disk | Premium SSD | Standard SSD | Standard HDD | | -------------- | ----------- | ----------- | ------------ | ------------ | | Max throughput | 2,000 MiB/s | 900 MiB/s | 750 MiB/s | 500 MiB/s | | Max IOPS | 160,000 | 20,000 | 6,000 | 2,000 | | Premium SSD sizes | P1\* | P2\* | P3\* | P4 | P6 | P10 | P15 | P20 | P30 | P40 | P50 | P60 | P70 | P80 | | ----------------- | ---- | ---- | ---- | --- | --- | --- | ----- | ----- | ----- | ----- | ----- | ------ | ------ | ------ | | Disk size in GiB | 4 | 8 | 16 | 32 | 64 | 128 | 256 | 512 | 1,024 | 2,048 | 4,096 | 8,192 | 16,384 | 32,767 | | IOPS per disk | 120 | 120 | 120 | 120 | 240 | 500 | 1,100 | 2,300 | 5,000 | 7,500 | 7,500 | 16,000 | 18,000 | 20,000 | ## cli ``` # Powershell Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\AzureCLI.msi; Start-Process msiexec.exe -Wait -ArgumentList '/I AzureCLI.msi /quiet' # Linux apt curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash # Linux all curl -L https://aka.ms/InstallAzureCli | bash docker run -it -v ${HOME}/.ssh:/root/.ssh mcr.microsoft.com/azure-cli az login az resource list ``` ### vm ``` az vm list --output table --show-details az vm stop/deallocate/start --resource-group linux --name ubuntu-2 ``` `deallocate` to stop billing & `disk update` ### disk Locally-redundant storage (LRS) ``` az disk list --resource-group linux --output table az disk update --resource-group linux --name ubuntu-2_OsDisk_1_74250d56f08040e1a48f38b9198148f7 --size-gb 32 # Reducing disk/snapshot size is not supported. create new snapshot/disks. az disk update --resource-group linux --name ubuntu-2_OsDisk_1_74250d56f08040e1a48f38b9198148f7 --sku Premium_LRS az disk create --location eastasia --sku Premium_LRS -g linux -n data-1 --size-gb 10 ``` ``` az vm disk attach -g linux --vm-name ubuntu-2 --name data-1 # --new ``` ### network lb #### inbound-nat-rule # cf * [Network ports](#network-ports) * [Rules](#rules) * [SSL/TLS](#ssltls) * [SSL Modes](#ssl-modes) * [Edge Certificates](#edge-certificates) * [Origin CA](#origin-ca) * [CNAME Flattening](#cname-flattening) * [Workers](#workers) * [TypeScript](#typescript) * [Wrangler (Workers CLI)](#wrangler-workers-cli) * [Workers as Reverse Proxy](#workers-as-reverse-proxy) * [Bulk origin override](#bulk-origin-override) * [Reflare](#reflare) * [Storage](#storage) * [R2 - S3 object](#r2---s3-object) * [KV - key-value](#kv---key-value) * [D1 - RDB](#d1---rdb) * [Durable Objects - Workers Paid plan](#durable-objects---workers-paid-plan) * [Queues - Workers Paid plan](#queues---workers-paid-plan) * [Paid](#paid) ## Network ports HTTP: 80 8080 | No caching: 8880 2052 2082 2086 2095\ HTTPS: 443 | No caching: 2053 2083 2087 2096 8443 ## Rules Settings - URL normalization: * [order](https://developers.cloudflare.com/rules/configuration-rules/#execution-order): Origin -> Cache -> Conf -> Single -> Page | Name | Free | Desc | | --------------------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------- | | [Transform Rules](https://developers.cloudflare.com/rules/transform/) | free 10 | no Regex support | | [Origin Rules](https://developers.cloudflare.com/rules/origin-rules/) | 10 | Override destination port | | [Cache Rules](https://developers.cloudflare.com/cache/about/cache-rules/) | 10 | cache properties of your HTTP requests | | [Configuration Rules](https://developers.cloudflare.com/rules/configuration-rules/) | 10 | [expressions](https://developers.cloudflare.com/firewall/cf-dashboard/edit-expressions/) / no Regex support | | [Single Redirects](https://developers.cloudflare.com/rules/url-forwarding/bulk-redirects/concepts/) | 5\~20 | | | [Page Rules](https://support.cloudflare.com/hc/en-us/articles/218411427) | 3 | require a "proxied" DNS record, highest priority rule at the top | ## SSL/TLS ### SSL Modes ``` Flexible: visitor - Edge cert - cf Full: cf - self signed certificate - server Full (strict): cf - trusted CA - server ``` ### Edge Certificates [Automatic HTTPS Rewrites](https://support.cloudflare.com/hc/en-us/articles/227227647) safely eliminates **mixed content** issues by rewriting insecure URLs dynamically from known secure hosts to their secure counterpart. [Page Rules - Always Use HTTPS](https://support.cloudflare.com/hc/en-us/articles/218411427#https) [Page Rules - Forwarding URL](https://support.cloudflare.com/hc/en-us/articles/200170536) ### Origin CA 15-years wildcard | visitor - Edge cert - cf - Origin CA - server： ``` PEM: Apache httpd and NGINX PKCS#7: Microsoft’s IIS or Apache Tomcat ``` ## CNAME Flattening * CNAME records normally can not be on the zone apex. We use CNAME flattening to make it possible. * DNS/Settings * speeds up CNAME resolution * CNAME flattening occurs by default ## Workers * Free Limits: * Examples: * Create site: * Cron: * KV: ### TypeScript ``` npm init cloudflare my-project worker-typescript ``` * Pages: ### Wrangler (Workers CLI) ``` npm install -g wrangler # nodejs v16.13.0+ wrangler login # install volta/nvm if error wrangler whoami # view permissions wrangler init && cd ``` `wrangler generate [name] [template]`: ``` npm start # [l] turn on/off local mode npm test npm run deploy || wrangler publish # 1st time: wait 1min ``` Workers > Overview # re-login web if no menu * Online dev: worker page > click `Quick edit` `wrangler dev`: `wrangler tail --format=pretty`: ## Workers as Reverse Proxy ### Bulk origin override ### Reflare ``` npm init cloudflare reflare-app https://github.com/xiaoyang-sde/reflare-template ``` ## Storage ### R2 - S3 object | Product | Free | Paid - Rates | | ------------------ | --------------------------- | ------------------------ | | Storage | 10 GB / month | $0.015 / GB-month | | Class A Operations | 1 million requests / month | $4.50 / million requests | | Class B Operations | 10 million requests / month | $0.36 / million requests | ### KV - key-value 1 GB free: Value size 25MB / 100,000 reads per day: ### D1 - RDB * only be charged for base storage plus any database operations performed ### Durable Objects - Workers Paid plan ### Queues - Workers Paid plan job queueing, batching and inter-Service (Worker to Worker) communication. ## Paid | Product | Fee | Desc | | -------------------------------------------------------------------------------- | ---------- | ---------------------------------------------------------------------- | | [Workers Paid plan](https://developers.cloudflare.com/workers/platform/pricing/) | $5 / month | separate from any other Cloudflare plan (Free, Professional, Business) | | Images | $5 / month | Store, resize, optimize and serve images at scale | | Stream | $5 / month | Live and on-demand video streaming in minutes | ## Community domain: # cf\_dev * [Free Lmits](#free-lmits) * [Worker](#worker) * [py](#py) * [js](#js) ## Free Lmits * CPU time per HTTP request 10 ms ## Worker * JavaScript + TypeScript * Python Workers * Rust ### py * beta @ 2026 March ### js ``` npm create cloudflare@latest -- qr-code-generator npx wrangler deploy ``` # gcp * [cli](#cli) * [Tasks](#tasks) * [GCS](#gcs) * [fuse](#fuse) ## cli ``` curl https://sdk.cloud.google.com | bash gcloud auth login ``` ## Tasks add project non-root ssh key: ## GCS ``` export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s` echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - sudo apt-get update sudo apt-get install -y gcsfuse mkdir gcsfuse gcsfuse ferro-asia gcsfuse ls gcsfuse ``` ## fuse # github * [Sponsorships](#sponsorships) * [Orgs](#orgs) ## Sponsorships ``` github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2] patreon: # Replace with a single Patreon username open_collective: # Replace with a single Open Collective username ko_fi: # Replace with a single Ko-fi username tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry liberapay: # Replace with a single Liberapay username issuehunt: # Replace with a single IssueHunt username otechie: # Replace with a single Otechie username lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] ``` ## Orgs * month-long celebration: # ibm\_bluemix * [cli](#cli) * [Container](#container) * [external services](#external-services) ## cli ``` curl -fsSL https://clis.ng.bluemix.net/install/linux | sh docker run --name bluemix \ -d --restart unless-stopped \ --privileged --net host \ reachlin/bluemix docker exec -it bluemix bash ``` ``` bx plugin list bx plugin install container-service -r Bluemix bx plugin update container-service -r Bluemix bx login -a https://api.eu-de.bluemix.net -sso bx cs region bx cs region-set eu-de bx cs clusters bx cs cluster-config mycluster bx cs workers mycluster # check public IP bx cs worker-update mycluster BLUEMIX_TRACE=path/to/trace.log # Append API request diagnostics to a log file BLUEMIX_API_KEY=api_key_value # API key to use during login ``` ## Container ``` NodePort service (free and standard clusters) LoadBalancer service (standard clusters only) Ingress (standard clusters only) ``` ![](https://dev-console.stage1.bluemix.net/docs/api/content/containers/images/networking.png?lang=en-US) ``` bx cs cluster-get mycluster # Master URL ``` [k8s related](/container/k8s) ## external services ``` bx service list name service plan bound apps last operation Monitoring-tz Monitoring lite create succeeded Visual Recognition-8p watson_vision_combined free create succeeded ``` # Pricing * [Always Free](#always-free) * [Azure](#azure) * [AWS](#aws) * [GCP](#gcp) * [Oracle](#oracle) * [Free CI/CD](#free-cicd) * [Github](#github) * [Jetbrains](#jetbrains) * [Compare](#compare) * [Instance](#instance) * [Traffic](#traffic) * [Price](#price) * [AWS](#aws-1) * [Azure](#azure-1) * [GCP](#gcp-1) * [IBM](#ibm) * [China](#china) * [VPS](#vps) * [Limited Free](#limited-free) * [AWS](#aws-2) * [Azure](#azure-2) * [IBM](#ibm-1) * [VPS](#vps-1) * [CDN](#cdn) * [China](#china-1) ## Always Free ### Azure ``` Free 2 million characters included for Translator Text API Free 5 GB per month analysis plus 31-day retention period with Log Analytics Azure Maps S0 account tiers offer 250,000 monthly map tile loads and 25,000 monthly service calls 10 web, mobile, or API apps with Azure App Service with 1 GB storage 1 million requests and 400,000 GBs of resource consumption with Azure Functions 100,000 operations for event publishing and delivery with Event Grid Free Azure Container service to cluster virtual machines 50,000 stored objects with Azure Active Directory with single sign-on (SSO) for 10 apps per user 50,000 active users per month (MAU) with Azure Active Directory B2C Free Azure Service Fabric to build microservice apps Unlimited nodes (server or PaaS instance) with Application Insights and 1 GB of telemetry data included per month Unlimited use of Azure DevTest Labs Machine Learning with 100 modules and 1 hour per experiment with 10 GB included storage Free policy assessment and recommendations with Azure Security Center Unlimited recommendations and best practices with Azure Advisor Azure IoT Central includes up to 5 devices with 50,000 monthly messages per device Free Azure IoT Hub edition includes 8,000 messages per day with 0.5 KB message meter size Free namespace and 1 million push notifications with Azure Notification Hubs 5 free low frequency activities with Azure Data Factory 50 MB storage for 10,000 hosted documents with Azure Cognitive Search including 3 indexes per service Unlimited Azure Batch usage for job scheduling and cluster management First 5 users free with Azure DevOps Free 500 minutes of job run time with Azure Automation Unlimited users and 5,000 catalog objects with Azure Data Catalog 30,000 transactions per month processing at 20 transactions per minute with Face API Free public load-balanced IP with Azure Load Balancer 5 GB of bandwidth for outbound data transfer with free unlimited inbound transfer 50 virtual networks free with Azure Virtual Network Unlimited inbound Inter-VNet data transfer ``` ### AWS ``` Lambda: 1M free requests per month Amazon DynamoDB 25 GB of storage ``` ### GCP ``` GAE 28 hours per day of "F" instances 9 hours per day of "B" instances 1 GB of egress per day GCE 1 non-preemptible e2-micro: Oregon: us-west1 / Iowa: us-central1 / South Carolina: us-east1 30 GB-months standard persistent disk | 1 GB network egress except CN/AU GCS: 5 GB + 1 GB network egress* Firestore DB: 1 GB storage ``` ### Oracle ``` 2 AMD based Compute VMs Standard.E2.1.Micro with 1/8 OCPU** and 1 GB memory each, in one availability domain | 1 OCPU = 2 x86 vCPU or 1 ARM vCPU Arm-based Ampere A1 cores and 24 GB of memory usable as 1 VM or up to 4 Standard.A1.Flex VMs with 3,000 OCPU hours and 18,000 GB hours per month 2 Block Volumes Storage, 200 GB total 10 GB Object Storage – Standard 10 GB Object Storage – Infrequent Access 10 GB Archive Storage Resource Manager: managed Terraform 5 OCI Bastions ``` ## Free CI/CD ### Github ### Jetbrains 2,000 Computation Credits per month : 1 CC/m = 2 vCPU + 7800MB memory : ## Compare ### Instance GCP: 20% vCPU + 0.60GB, Free - $4.09\ Azure: 1vCPU + 0.5 GiB, $1.45/m(West US 2, 3-year) Amazon: 5% vCPU + 0.5GB, 3 Year Reserved Instance $69 ### Traffic Amazon: 1GB to 10 TB / month $0.090 per GB, Singapore to China $0.120 per GB\ Google: $0.12 - $0.23 / GB, Singapore to China $0.23\ Microsoft: 5 GB - 10 TB 2 /Month $0.087 - $0.181 per GB, Japan to China $0.138 per GB ## Price ### AWS Traffic: ### Azure ``` 5 GB - 10 TB /Month: $0.12 per GB ``` ``` BASIC (CLASSIC) + Dynamic IP address + First Cloud Service VIP = free ``` ### GCP ### IBM ### China (Compute)\ (HDD) ### VPS ## Limited Free ### AWS ### Azure $200: ``` 15 GB outbound ``` ### IBM ### VPS DigitalOcean $20: ## CDN ### China JD 动态加速 赠送流量： qn 10G/m/http： tx 新户100G/6m： ali： hw： dcloud free: 云函数： 资源使用量：1000GBs/月 调用次数：1.5万次 出网流量：1GB 云数据库： 集合数量 100 / 索引数量 400 容量：2GB 读操作数：0.05万次/天 写操作数：0.03万次/天 云存储： 容量：5GB 下载操作次数：0.2万次 上传操作次数：0.1万次 CDN流量：1GB 静态网站托管： 容量：5GB CDN流量：1GB # COS ## coscli ``` wget https://cosbrowser.cloud.tencent.com/software/coscli/coscli-linux-amd64 coscli config init # ~/.cos.yaml coscli ls # buckets coscli sync dir1 cos://bucket1 # upload coscli sync cos://bucket1 dir1 # download ``` # container ## cilium HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP ## LXD # docker * [Web UI](#web-ui) * [Scripts](#scripts) * [Dockerfile code snippets](#dockerfile-code-snippets) * [CMD and ENTRYPOINT](#cmd-and-entrypoint) * [apt](#apt) * [alpine](#alpine) * [tini](#tini) * [S6 - a process supervisor](#s6---a-process-supervisor) * [badger](#badger) * [Storage](#storage) * [btrfs issue](#btrfs-issue) * [Network](#network) * [macvlan/ipvlan](#macvlanipvlan) * [plugins](#plugins) * [DHCP](#dhcp) * [Commands](#commands) * [build](#build) * [container/image operations](#containerimage-operations) * [cp](#cp) * [run](#run) * [X11 Forwarding](#x11-forwarding) * [container update](#container-update) * [Clean up](#clean-up) * [Detach](#detach) * [Config](#config) * [Mirrors](#mirrors) * [Proxy](#proxy) * [Swarm](#swarm) * [OS](#os) * [CoreOS](#coreos) * [boot2docker](#boot2docker) * [Windows/Mac](#windowsmac) * [Automated builds](#automated-builds) ## Web UI ## Scripts ``` source /dev/stdin <<< "$(curl -sSL https://raw.githubusercontent.com/fzinfz/scripts/master/linux/docker.sh)" ``` ## Dockerfile code snippets ### CMD and ENTRYPOINT ``` ENTRYPOINT [“exec_entry”, “p1_entry”] CMD [“exec_cmd”, “p1_cmd”] => exec_entry p1_entry exec_cmd p1_cmd ``` ### apt ``` RUN apt update && apt install -y --no-install-recommends && rm -r /var/lib/apt/lists/* ``` ### alpine ``` # install pip3 RUN wget https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && rm get-pip.py RUN apk add --no-cache --virtual .build-deps \ curl ca-certificates jq \ && apk del .build-deps ``` ### tini ``` docker run --init ``` ### S6 - a process supervisor ## badger ## Storage ``` When possible, overlay2 is the recommended storage driver. Supported backing filesystems: xfs with ftype=1, ext4 ( where /var/lib/docker/ is located ) ``` ### btrfs issue ## Network none/bridge/host/overlay/{belows}: ### macvlan/ipvlan * Ipvlan：All sub-interfaces share parent’s MAC | vs Macvlan * Ipvlan L3 mode：Each sub-interface has to be configured with a different subnet * Macvlan and ipvlan cannot be used on the same parent interface at the same time. * IPvlan L2 mode trunking is the same as Macvlan with regard to gateways and L2 path isolation. * `--internal`: ( off `-o parent=` ) * if no `--gateway`: gw for `--subnet=192.168.1.0/24` will be 192.168.1.1 To access host, check [/nw/openwrt](https://github.com/fzinfz/book/blob/master/nw/openwrt/README.md) ### plugins ### DHCP * * ## Commands ### build ``` docker build [-f Dockerfile.custom] [--target multi-stage] Dockerfile-Root-Folder docker build - < Dockerfile # no context, local ADD not working curl example.com/remote/Dockerfile | docker build -f - . Get-Content Dockerfile | docker build - # Powershell docker build -f ctx/Dockerfile http://server/ctx.tar.gz docker build https://github.com/user/repo.git ``` | Build Syntax Suffix | Commit Used | Build Context Used | | ---------------------------- | ------------------- | ------------------ | | myrepo.git#mytag:myfolder | refs/tags/mytag | /myfolder | | myrepo.git#mybranch:myfolder | refs/heads/mybranch | /myfolder | Squashing does not destroy any existing image, rather it creates a new image. ### container/image operations ``` docker image tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG] docker export container_name > container.tar docker import [OPTIONS] file|URL|- [REPOSITORY[:TAG]] docker save image_name > image.tar docker load < image.tar[.gz] docker save python | ssh -C 192.168.88.72 docker load ``` ### cp ``` docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|- docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH ``` ### run ``` --user=[ user | user:group | uid | uid:gid | user:gid | uid:group ] -m, --memory="" -c, --cpu-shares=0 CPU shares (relative weight) --dns=[] : Set custom dns servers for the container --network="bridge" : Connect a container to a network 'bridge': create a network stack on the default Docker bridge 'none': no networking 'container:': reuse another container's network stack 'host': use the Docker host network stack '|': connect to a user-defined network --network-alias=[] : Add network-scoped alias for the container --add-host="" : Add a line to /etc/hosts (host:IP) --mac-address="" : Sets the container's Ethernet device's MAC address --ip="" : Sets the container's Ethernet device's IPv4 address --link-local-ip=[] : Sets one or more container's Ethernet device's link local IPv4/IPv6 addresses --read-only ：prohibiting writes to locations other than the specified volumes Volume labels `:z` => shared `:Z` => private `--entrypoint` will clear out `CMD` echo test | docker run --rm -i alpine cat docker run --security-opt seccomp:unconfined # may fix chromium start error ``` #### X11 Forwarding ``` --env="DISPLAY" --volume="$HOME/.Xauthority:/root/.Xauthority:rw" ``` ### container update docker container update \[OPTIONS] CONTAINER \[CONTAINER...] --cpus="1.5" # one and a half of the CPUs --cpu-shares , -c --memory , -m Memory limit --memory-reservation Memory soft limit --restart ### Clean up ``` docker container prune docker system prune # Remove unused data ``` ### Detach ``` Ctrl+p & Ctrl+q ``` ## Config ``` /etc/docker/daemon.json # delete `,` & `#...` # `dockerd` for debugging: https://docs.docker.com/engine/admin/ { "live-restore": true, # containers remain running if daemon unavailable "graph": "/data/docker-fs", "storage-driver": "overlay2", } ``` ### Mirrors CN: LAN: ### Proxy 23.0+ : * daemon configuration file : /etc/docker/daemon.json * * command-line options : ``` mkdir -p /etc/systemd/system/docker.service.d cat > /etc/systemd/system/docker.service.d/http-proxy.conf << EOF [Service] Environment="HTTP_PROXY=http://127.0.0.1:1081" Environment="HTTPS_PROXY=http://127.0.0.1:1081" Environment="NO_PROXY=localhost,127.0.0.1,192.168.*.*,172.16.*.*,100.*.*.*" EOF systemctl daemon-reload && systemctl restart docker systemctl show --property=Environment docker ``` ## Swarm TCP port 2377 for cluster management communications TCP and UDP port 7946 for communication among nodes TCP and UDP port 4789 for overlay network traffic --opt encrypted => protocol 50 (ESP) is open ``` docker swarm init --advertise-addr 10.2.0.1 docker swarm join-token manager docker swarm join-token worker docker swarm init --force-new-cluster # without losing data docker node ls docker node update --label-add server=s1 st netstat -lntup | egrep '2377|7946|4789|50' docker service create --name nginx -p 8080:80 --replicas 3 nginx docker service create --name nginx -p 80:80 -p 443:443 --network web --mode global nginx docker service ls docker service ps nginx docker inspect | grep Err ``` ``` docker network create \ --driver overlay \ --subnet 10.66.3.0/24 \ --opt encrypted \ web docker network ls # node IP ip addr | grep -P -o '\d+\.\d+\.\d+\.\d+(?=/24)' # service VIP ip addr | grep -P -o '\d+\.(?!255)\d+\.\d+\.\d+(?=/32)' ``` ## OS ### CoreOS ``` vi /etc/coreos/update.conf update_engine_client -update ``` ### boot2docker Lightweight Linux for Docker ``` echo EXTRA_ARGS="--foo=bar" >> /var/lib/boot2docker/profile ``` ## Windows/Mac ## Automated builds Docker Hub | $5+/m: Github Actions | 2k min/m: Jetbrains Space | # Kubernetes * [Docs](/container/k8s#docs) * [Components](/container/k8s#components) * [Creating HA clusters](/container/k8s#creating-ha-clusters) * [Objects](/container/k8s#objects) * [namespaces](/container/k8s#namespaces) * [controllers](/container/k8s#controllers) * [ReplicaSet](/container/k8s#replicaset) * [Replication Controller](/container/k8s#replication-controller) * [Deployment controller](/container/k8s#deployment-controller) * [StatefulSet](/container/k8s#statefulset) * [DaemonSet](/container/k8s#daemonset) * [Job & CronJob](/container/k8s#job--cronjob) * [kubectl](/container/k8s#kubectl) * [dashboard admin](/container/k8s#dashboard-admin) * [expose](/container/k8s#expose) * [token](/container/k8s#token) * [Tutorial](/container/k8s#tutorial) * [yaml](/container/k8s#yaml) * [syntax](/container/k8s#syntax) * [env](/container/k8s#env) * [Command and Arguments](/container/k8s#command-and-arguments) * [generator](/container/k8s#generator) * [Verbosity](/container/k8s#verbosity) * [minikube](/container/k8s#minikube) * [Bare metal](/container/k8s#bare-metal) * [KVM](/container/k8s#kvm) * [kops](/container/k8s#kops) * [Create a Cluster](/container/k8s#create-a-cluster) * [Persistent Volumes](/container/k8s#persistent-volumes) * [helm - package manager](/container/k8s#helm---package-manager) * [WebUI](/container/k8s#webui) * [Hub](/container/k8s#hub) ## Docs ``` kubectl run --image=nginx nginx-app --port=80 --env="DOMAIN=cluster" # deployment "nginx-app" created kubectl expose deployment nginx-app --port=80 --name=nginx-http # service "nginx-http" exposed kubectl exec nginx-app-5jyvm -- cat /etc/hostname kubectl get deployment kubectl get pods -a kubectl logs kubectl version --short ``` ## Components ## Creating HA clusters ## Objects persistent entities\ All objects in REST API are identified by a Name(such as /api/v1/pods/some-name) and a UID.\ For non-unique user-provided attributes, Kubernetes provides labels and annotations. ### namespaces multiple virtual clusters backed by the same physical cluster ``` kubectl get namespaces NAME STATUS AGE default Active 59d ibm-cert-store Active 59d ibm-system Active 59d kube-public Active 59d kube-system Active 59d default: for objects with no other namespace kube-system: created by the Kubernetes system kube-public: readable by all users. reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster. The public aspect of this namespace is only a convention, not a requirement ``` ## controllers ### ReplicaSet ReplicaSet is the next-generation Replication Controller.\ ReplicaSet supports the new set-based selector requirements ``` kubectl get pods -l 'environment,environment notin (frontend)' ``` ### Replication Controller only supports equality-based selector requirements. ``` kubectl get pods -l environment=production,tier=frontend ``` ### Deployment controller \ provides declarative updates for Pods and ReplicaSets. ### StatefulSet workload API object used to manage stateful applications.\ Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. ``` Stable, unique network identifiers. Stable, persistent storage. Ordered, graceful deployment and scaling. Ordered, graceful deletion and termination. Ordered, automated rolling updates. ``` ### DaemonSet ensures that all (or some) Nodes run a copy of a Pod. ### Job & CronJob ## kubectl ``` curl -LO https://storage.googleapis.com/kubernetes-release/release/$( \ curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt \ )/bin/linux/amd64/kubectl chmod +x kubectl && mv kubectl /usr/local/bin/ sudo snap install kubectl --classic kubectl get nodes export KUBECONFIG=/path/to/kube-config-mil01-mycluster.yml # kubectl proxy --address='0.0.0.0' --accept-hosts='.*' --port=8080 kubectl proxy --address=$IP_Private --accept-hosts='^.*$' # http://...:8080/ui ``` ## dashboard admin ## expose ``` kubectl port-forward redis-master 6379:6379 ``` `kubectl expose -h` ``` pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs) # Create a service for a replicated nginx, which serves on port 80 and connects to the containers on port 8000. kubectl expose rc nginx --port=80 --target-port=8000 # Create a service for a replication controller identified by type and name specified in "nginx-controller.yaml", which serves on port 80 and connects to the containers on port 8000. kubectl expose -f nginx-controller.yaml --port=80 --target-port=8000 # Create a service for a pod valid-pod, which serves on port 444 with the name "frontend" kubectl expose pod valid-pod --port=444 --name=frontend # Create a second service based on the above service, exposing the container port 8443 as port 443 with the name "nginx-https" kubectl expose service nginx --port=443 --target-port=8443 --name=nginx-https # Create a service for a replicated streaming application on port 4100 balancing UDP traffic and named 'video-stream'. kubectl expose rc streamer --port=4100 --protocol=udp --name=video-stream # Create a service for a replicated nginx using replica set, which serves on port 80 and connects to the containers on port 8000. kubectl expose rs nginx --port=80 --target-port=8000 # Create a service for an nginx deployment, which serves on port 80 and connects to the containers on port 8000. kubectl expose deployment nginx --port=80 --target-port=8000 ``` ## token ``` kubectl describe secret kubectl config view -o jsonpath='{.users[0].user.auth-provider.config.id-token}' ``` ## Tutorial ``` kubectl run nginx --image=nginx kubectl create deployment nginx --image nginx # do the same thing as above kubectl expose deployment/nginx --name=nginx --type=NodePort --port=80 --target-port=80 kubectl describe services nginx kubectl run hello-world --replicas=2 \ --labels="run=load-balancer-example" \ --image=gcr.io/google-samples/node-hello:1.0 \ --port=8080 kubectl get deployments hello-world kubectl describe deployments hello-world kubectl get replicasets kubectl describe replicasets kubectl expose deployment hello-world --type=LoadBalancer --name=my-service kubectl describe services my-service kubectl get services ``` ## yaml Release 1.8: `apps/v1beta1` -> `apps/v1beta2`; 1.9： -> `apps/v1` ``` kubectl create -f nginx.yaml kubectl replace -f nginx.yaml # updates from another source will be lost kubectl delete -f nginx.yaml -f redis.yaml ``` `kubectl apply` supports multiple writers to the same object. ``` kubectl apply -f configs/ kubectl apply -R -f configs/ # Recursively kubectl get -f https://example.com/x.yaml -o yaml # print kubectl get / -o yaml --export > _.yaml # export ``` ### syntax #### env #### Command and Arguments | Description | Docker field name | Kubernetes field name | | ----------------------------------- | ----------------- | --------------------- | | The command run by the container | Entrypoint | command | | The arguments passed to the command | Cmd | args | ``` spec: containers: - name: command-demo-container image: debian command: ["printenv"] args: ["HOSTNAME", "KUBERNETES_PORT"] command: - sh - -c - while true; do sleep 1; done ``` Example： ### generator ## Verbosity | Verbosity | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --v=0 | Generally useful for this to ALWAYS be visible to an operator. | | --v=1 | A reasonable default log level if you don’t want verbosity. | | --v=2 | Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level for most systems. | | --v=3 | Extended information about changes. | | --v=4 | Debug level verbosity. | | --v=6 | Display requested resources. | | --v=7 | Display HTTP request headers. | | --v=8 | Display HTTP request contents. | ## minikube ``` curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/v1.8.0/bin/linux/amd64/kubectl chmod +x minikube kubectl && mv minikube kubectl /usr/local/bin/ ``` ### Bare metal ``` minikube start --vm-driver=none && minikube config set vm-driver none minikube logs ``` ### KVM ``` sudo usermod -a -G libvirt $(whoami) newgrp libvirt minikube config set vm-driver kvm ``` ## kops \ kubectl for clusters ## Create a Cluster ## Persistent Volumes ``` apiVersion: v1 kind: PersistentVolume metadata: name: pv0003 spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: slow mountOptions: - hard - nfsvers=4.1 nfs: path: /tmp server: 172.17.0.2 ``` | Volume Plugin | ReadWriteOnce | ReadOnlyMany | ReadWriteMany | | -------------------- | ------------- | ------------ | ---------------------------------- | | AWSElasticBlockStore | ✓ | - | - | | AzureFile | ✓ | ✓ | ✓ | | AzureDisk | ✓ | - | - | | CephFS | ✓ | ✓ | ✓ | | Cinder | ✓ | - | - | | FC | ✓ | ✓ | - | | FlexVolume | ✓ | ✓ | - | | Flocker | ✓ | - | - | | GCEPersistentDisk | ✓ | ✓ | - | | Glusterfs | ✓ | ✓ | ✓ | | HostPath | ✓ | - | - | | iSCSI | ✓ | ✓ | - | | PhotonPersistentDisk | ✓ | - | - | | Quobyte | ✓ | ✓ | ✓ | | NFS | ✓ | ✓ | ✓ | | RBD | ✓ | ✓ | - | | VsphereVolume | ✓ | - | - (works when pods are collocated) | | PortworxVolume | ✓ | - | ✓ | | ScaleIO | ✓ | ✓ | - | | StorageOS | ✓ | - | - | ## helm - package manager ### WebUI ``` helm repo add monocular https://kubernetes-helm.github.io/monocular helm install monocular/monocular kubectl get ingress ``` \ navigate and search Helm Charts. ### Hub # lxc-pve * [Images](#images) * [Proxmox Container Toolkit](#proxmox-container-toolkit) * [share host folder](#share-host-folder) * [Unprivileged](#unprivileged) ## Images ``` pveam update # daily through the `pve-daily-update` timer pveam available --section system pveam list local pveam download local {}.tar.gz # WebUI `CT Templates`: click `Templates` button ``` ## Proxmox Container Toolkit ``` pct list pct exec 201 -- bash -lc "volta list all" # `-l` = login shell, loads `.profile`/`.bash_profile` ``` ## share host folder ``` pct set 201 -mp1 /data,mp=/data ``` ## Unprivileged ``` apt install acl setfacl -m u:100000:rwx /host/folder/path ``` # podman * [Config](#config) * [CLI](#cli) * [network](#network) * [Netavark](#netavark) * [CNI](#cni) * [DHCP](#dhcp) * [create](#create) ## Config V2: /etc/containers/registries.conf ``` unqualified-search-registries = ["docker.io"] ``` ## CLI ``` podman search docker.io/KEYWORD ``` ## network * Netavark is the default network backend and was added in Podman v4.0 * CNI will be deprecated /etc/containers/containers.conf ``` [network] network_backend="" # cni / netavark ``` ``` podman network inspect podman ``` ### Netavark * Better IPv6 support * Improved support for containers in multiple networks * Improved performance ### CNI cni\_config\_dir in containers.conf: /etc/cni/net.d DHCP: ### create `--driver` * bridge * macvlan * ipvlan `--ipam-driver` * dhcp: not yet supported with netavark | For CNI the dhcp plugin needs to be activated before. * host-local * none # db * [Couchbase vs CouchDB](#couchbase-vs-couchdb) * [API Query](#api-query) * [key-value](#key-value) * [Time Series DBs](#time-series-dbs) * [TimescaleDB](#timescaledb) * [Riak TS](#riak-ts) * [Firebase](#firebase) * [Supabase](#supabase) ## Couchbase vs CouchDB || Couchbase Server|Apache CouchDB| |---|---| |Topology|Distributed|Replicated| |Automatic failover|Yes|No| |Integrated cache|Yes|No| |Memcached compatible|Yes|No| |Query language|Yes, N1QL (SQL for JSON)|No| ## API Query ## key-value ## Time Series DBs Why relational database instead of NoSQL: Elastic, InfluxDB, MongoDB, Cassandra, Couchbase, Graphite, Prometheus, ClickHouse, OpenTSDB, DalmatinerDB, KairosDB, RiakTS. ## TimescaleDB packaged as a PostgreSQL extension ``` docker run -d --name timescaledb -p 5432:5432 timescale/timescaledb ``` ``` CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE; SELECT create_hypertable('conditions', 'time'); -- backgroud: CREATE INDEX conditions_time_idx ON public.conditions USING btree ("time" DESC) TABLESPACE pg_default; -- additionally partition the data on another -- dimension (what we call 'space partitioning'). -- E.g., to partition `location` into 4 partitions: SELECT create_hypertable('conditions', 'time', 'location', 4); SELECT time_bucket('5 minutes', time) AS time_range, location, COUNT(*), MAX(temperature) AS max_temp, MAX(humidity) AS max_hum FROM conditions WHERE time > NOW() - interval '3 hours' GROUP BY time_range, location ORDER BY time_range DESC, max_temp DESC; time_range | location | count | max_temp | max_hum ------------------------+----------+-------+----------+--------- 2018-02-23 17:00:00+00 | office | 3 | 70 | 50 2018-02-23 16:35:00+00 | garage | 1 | 77 | 65.2 2018-02-23 16:35:00+00 | office | 2 | 70.1 | 50.1 2018-02-23 16:35:00+00 | basement | 1 | 66.5 | 60 2018-02-23 16:25:00+00 | office | 1 | 70 | 50 (5 rows) ``` ## Riak TS Riak TS is a distributed NoSQL key/value store optimized for time series data. ## Firebase Realtime NoSQL: ## Supabase Firebase alternative, PostgreSQL * OSS: * # InfluxDB * [Data Format](#data-format) * [Client](#client) * [cli](#cli) * [Python](#python) * [Flux Query](#flux-query) * [difference()](#difference) ## Data Format ``` [,=...] \ =[,=...] \ [unix-nano-timestamp] ``` ## Client ### cli ``` for c in version config ; do docker exec -it influxdb influx $c ; done docker exec -it influxdb influx bucket list --org $INFLUXDB_ORG --token $INFLUXDB_TOKEN ``` ### Python ## Flux Query ### difference() # loki * [API](/db/loki#api) * [Query](/db/loki#query) * [Python](/db/loki#python) ## API ### Query ``` curl -G -s "http://localhost:3100/loki/api/v1/query" \ --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' \ | jq ``` ## Python # MySQL & MariaDB * [Storage Engines](/db/mysql#storage-engines) * [MySQL Cluster](/db/mysql#mysql-cluster) * [JSON](/db/mysql#json) * [MySQL 5.7+](/db/mysql#mysql-57) * [MariaDB 10.2+](/db/mysql#mariadb-102) ## Storage Engines \ Compare: Table 15.1 Storage Engines Feature Summary | Feature | MyISAM | Memory | InnoDB | Archive | NDB | | ------------------------------------- | ------- | ----------- | ------- | ------- | ------- | | Storage limits | 256TB | RAM | 64TB | None | 384EB | | Transactions | No | No | Yes | No | Yes | | Locking granularity | Table | Table | Row | Row | Row | | MVCC | No | No | Yes | No | No | | Geospatial data type support | Yes | No | Yes | Yes | Yes | | Geospatial indexing support | Yes | No | Yes\[a] | No | No | | B-tree indexes | Yes | Yes | Yes | No | No | | T-tree indexes | No | No | No | No | Yes | | Hash indexes | No | Yes | No\[b] | No | Yes | | Full-text search indexes | Yes | No | Yes\[c] | No | No | | Clustered indexes | No | No | Yes | No | No | | Data caches | No | N/A | Yes | No | Yes | | Index caches | Yes | N/A | Yes | No | Yes | | Compressed data | Yes\[d] | No | Yes\[e] | Yes | No | | Encrypted data\[f] | Yes | Yes | Yes | Yes | Yes | | Cluster database support | No | No | No | No | Yes | | Replication support\[g] | Yes | Limited\[h] | Yes | Yes | Yes | | Foreign key support | No | No | Yes | No | Yes\[i] | | Backup / point-in-time recovery\[j] | Yes | Yes | Yes | Yes | Yes | | Query cache support | Yes | Yes | Yes | Yes | Yes | | Update statistics for data dictionary | Yes | Yes | Yes | Yes | Yes | ``` CREATE TABLE t1 (i INT) ENGINE = INNODB; InnoDB: default in MySQL 5.7. transaction-safe (ACID compliant)：commit, rollback, and crash-recovery row-level locking (without escalation to coarser granularity locks) Oracle-style consistent nonlocking reads clustered indexes to reduce I/O for common queries based on primary keys FOREIGN KEY referential-integrity constraints MyISAM: Table-level locking. often used in read-only or read-mostly workloads. Merge: logically group a series of identical MyISAM tables and reference them as one object. Memory or HEAP: Its use cases are decreasing: InnoDB with its buffer pool memory area NDBCLUSTER provides fast key-value lookups for huge distributed data sets NDB or NDBCLUSTER: highest possible degree of uptime and availability. Federated: link separate MySQL servers to create one logical database CSV: Its tables are really text files with comma-separated values. Archive: compact, unindexed tables Blackhole: not store data, Queries always return an empty set. can be used in replication configurations Example: illustrates how to begin writing new storage engines. ``` ## MySQL Cluster \ ## JSON ### MySQL 5.7+ ``` CREATE TABLE t1 (jdoc JSON); INSERT INTO t1 VALUES('{"key1": "value1", "key2": "value2"}'); SELECT JSON_ARRAY('a', 1, NOW()); SELECT JSON_OBJECT('key1', 1, 'key2', 'abc'); SELECT JSON_MERGE('["a", 1]', '{"key": "value"}'); SET @j = JSON_OBJECT('key', 'value'); SELECT @j; # escape quote character '{"mascot": "... \\"Sakila\\"."}' JSON_OBJECT("mascot", "... \"Sakila\".") JSON_OBJECT('mascot', '... "Sakila".') # NO_BACKSLASH_ESCAPES # JSON values is case sensitive SELECT CAST('null' AS JSON); # `null`, `true`, and `false` always lowercase SELECT col->"$.mascot" FROM qtest; # "... \"Sakila\"." SELECT sentence->>"$.mascot" FROM facts; # ... "Sakila". ORDER BY CAST(JSON_EXTRACT(jdoc, '$.id') AS UNSIGNED) ``` ### MariaDB 10.2+ ``` CREATE TABLE products(id INTEGER NOT NULL PRIMARY KEY AUTO_INCREMENT, name VARCHAR(255) NOT NULL, price DECIMAL(9,2) NOT NULL, stock INTEGER NOT NULL, attr VARCHAR(1024), CHECK (attr IS NULL OR JSON_VALID(attr))); INSERT INTO products VALUES(NULL, 'Blouse', 17, 15, '{"colour": "white"}'); UPDATE products SET attr = JSON_REPLACE(attr, '$.colour', 'red') WHERE name = 'Blouse'; ALTER TABLE products ADD attr_colour VARCHAR(32) AS (JSON_VALUE(attr, '$.colour')); CREATE INDEX products_attr_colour_ix ON products(attr_colour); EXPLAIN SELECT * FROM products WHERE attr_colour = 'white'; # verify index ``` # Oracle * [Storage](/db/oracle#storage) * [Tools](/db/oracle#tools) * [User](/db/oracle#user) * [Create Table](/db/oracle#create-table) * [Create Procedure](/db/oracle#create-procedure) ## Storage ![](https://docs.oracle.com/cloud/latest/db112/CNCPT/img/cncpt227.gif) ![](https://docs.oracle.com/cloud/latest/db112/CNCPT/img/cncpt027.gif) A `tablespace` is a logical storage container for segments. Segments are database objects, such as tables and indexes, that consume storage space.\ A `segment` is a set of extents allocated for a specific database object, such as a table. Each segment belongs to one and only one tablespace.\ An `extent` is a set of logically contiguous data blocks allocated for storing a specific type of information.\ One `logical data block` corresponds to a specific number of bytes of physical disk space, for example, 2 KB. ## Tools ## User ``` ALTER USER hr ACCOUNT UNLOCK; ALTER USER hr IDENTIFIED BY hr_password; ``` ## Create Table ![](https://docs.oracle.com/cd/B28359_01/server.111/b28286/img/create_table.gif) ``` CREATE TABLE "HR"."COUNTRIES" ( "COUNTRY_ID" CHAR(2 BYTE) CONSTRAINT "COUNTRY_ID_NN" NOT NULL ENABLE, ``` ![](https://docs.oracle.com/cd/B19306_01/server.102/b14200/img/inline_constraint.gif)\ ``` "COUNTRY_NAME" VARCHAR2(40 BYTE), "REGION_ID" NUMBER, CONSTRAINT "COUNTRY_C_ID_PK" PRIMARY KEY ("COUNTRY_ID") ENABLE, CONSTRAINT "COUNTR_REG_FK" FOREIGN KEY ("REGION_ID") ``` ![](https://docs.oracle.com/cd/B19306_01/server.102/b14200/img/out_of_line_constraint.gif) ``` REFERENCES "HR"."REGIONS" ("REGION_ID") ENABLE ``` ![](https://docs.oracle.com/cd/B19306_01/server.102/b14200/img/references_clause.gif) ![](https://docs.oracle.com/cd/B19306_01/server.102/b14200/img/constraint_state.gif) ``` ) ORGANIZATION INDEX -- index-organized table. ``` ORGANIZATION: the order in which the data rows of the table are stored. * HEAP: the data rows of table are stored in no particular order. This is the *default*. * INDEX: table is created as an index-organized table. In an index-organized table, the data rows are held in an index defined on the primary key for the table. * EXTERNAL: table is a read-only table located outside the database. NOCOMPRESS -- whether to compress data segments to reduce disk use PCTFREE 10 -- NOCOMPRESS use the PCTFREE default value of 10, to maximize compress while still allowing for some future DML changes to the data INITRANS 2 -- Specify the initial number of concurrent transaction entries allocated within each data block allocated to the database object. MAXTRANS 255 -- deprecated. LOGGING -- a database object will be logged in the redo log file ``` STORAGE( INITIAL 65536 -- the size of the first extent of the object. allocates space when you create the schema object. NEXT 1048576 -- in bytes the size of the next extent to be allocated to the object. MINEXTENTS 1 -- In locally managed tablespaces, determine the initial segment size in conjunction with PCTINCREASE, INITIAL and NEXT MAXEXTENTS 2147483645 -- valid only for objects in dictionary-managed tablespaces PCTINCREASE 0 -- Oracle recommends a setting of 0 as a way to minimize fragmentation and avoid the possibility of very large temporary segments during processing. FREELISTS 1 -- each free list group contains one free list FREELIST GROUPS 1 -- In tablespaces with manual segment-space management, statically partition the segment free space in an Oracle Real Application Clusters environment. BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT ) ``` ![](https://docs.oracle.com/cd/B28359_01/server.111/b28286/img/storage_clause.gif)\ ``` TABLESPACE "USERS" PCTTHRESHOLD 50; -- when an overflow segment is being used, defines the maximum size of the portion of the row that is stored in the index block, as a percentage of block size. 1–50. The default is 50. ``` ## Create Procedure ![](https://docs.oracle.com/cd/B19306_01/server.102/b14200/img/create_procedure.gif) ``` CREATE PROCEDURE find_root ( x IN REAL ) IS LANGUAGE C ``` Use the call\_spec to map a Java or C method name, parameter types, and return type to their SQL counterparts.\ ![](https://docs.oracle.com/cd/B28359_01/appdev.111/b28370/img/call_spec.gif)\ ![](https://docs.oracle.com/cd/B28359_01/appdev.111/b28370/img/java_declaration.gif) ``` NAME c_find_root LIBRARY c_utils PARAMETERS ( x BY REFERENCE ); ``` ![](https://docs.oracle.com/cd/B28359_01/appdev.111/b28370/img/c_declaration.gif) ``` create or replace PROCEDURE add_job_history ( p_emp_id job_history.employee_id%type , p_start_date job_history.start_date%type , p_end_date job_history.end_date%type , p_job_id job_history.job_id%type , p_department_id job_history.department_id%type ) IS BEGIN INSERT INTO job_history (employee_id, start_date, end_date, job_id, department_id) VALUES(p_emp_id, p_start_date, p_end_date, p_job_id, p_department_id); END add_job_history; ``` # PostgreSQL * [Info](/db/pgsql#info) * [Config](/db/pgsql#config) * [pgadmin](/db/pgsql#pgadmin) * [Commands](/db/pgsql#commands) * [user](/db/pgsql#user) * [db](/db/pgsql#db) ## Info ``` SELECT version(); SELECT now(); ``` ## Config ``` SHOW config_file; -- /var/lib/postgresql/data/postgresql.conf SHOW data_directory; -- /var/lib/postgresql/data SHOW hba_file; -- /var/lib/postgresql/data/pg_hba.conf -- host-based authentication -- https://www.postgresql.org/docs/current/static/auth-pg-hba-conf.html SHOW ident_file; -- /var/lib/postgresql/data/pg_ident.conf -- user name mapping: SYSTEM-USERNAME PG-USERNAME SHOW external_pid_file; ``` ## pgadmin ``` docker run -p 80:80 \ -e "PGADMIN_DEFAULT_EMAIL=user@domain.com" \ -e "PGADMIN_DEFAULT_PASSWORD=SuperSecret" \ -d dpage/pgadmin4 ``` DB->'Schemas'->'public'->'Tables'->right click table->'View/Edit Data'->'All Rows' ## Commands ## user ``` SELECT usename FROM pg_user; CREATE USER fzinfz; \password fzinfz ALTER USER fzinfz WITH SUPERUSER; \du -- list user/role ``` ## db ``` CREATE database tutorial; \c tutorial psql -U postgres -h localhost -d tutorial ``` # dev * [Repo](#repo) * [GraphQL](#graphql) * [Markdown](#markdown) * [Parser](#parser) * [Converter](#converter) * [Markdown + LaTeX](#markdown--latex) * [reStructuredText](#restructuredtext) * [OCR](#ocr) * [Offline](#offline) * [Windows Universal](#windows-universal) * [Excel](#excel) * [Formula](#formula) * [CSharp](#csharp) * [CSharp](#csharp-1) * [.Net Core](#net-core) * [InPage Search](#inpage-search) * [My Notes](#my-notes) ## Repo | byGrok | **Monolith** | **Monorepo** | **Polyrepo** (Multi-repo) | | --------------------- | ----------------------------- | -------------------------------------- | ---------------------------- | | **Definition** | Single deployable application | Single Git repository | Multiple Git repositories | | **Code organization** | One codebase, tightly coupled | Multiple projects/packages in one repo | One project/package per repo | **Quick rule of thumb** * **Monolith** → simple app, early stage * **Monorepo** → many related projects that share a lot of code * **Polyrepo** → independent projects/teams/services ## GraphQL a query language and execution engine tied to any backend service. ``` enum Episode { NEWHOPE, EMPIRE, JEDI } interface Character { id: String name: String friends: [Character] appearsIn: [Episode] } type Human implements Character { id: String name: String friends: [Character] appearsIn: [Episode] homePlanet: String } ``` ## Markdown ### Parser ``` var converter = new showdown.Converter(); var md = '- works in the server and in the **browser**'; var html = converter.makeHtml(md); ``` ### Converter Text/HTML table: CSV/WIKI table: HTML(non table): ## Markdown + LaTeX ## reStructuredText * .md -> .rst: * table -> .rst: ## OCR ### Offline text on screen: ### Windows Universal ## Excel ### Formula ``` =LEFT(B5,FIND("_",B5)-1) =RIGHT(B5,LEN(B5)-FIND("_",B5)) ``` ### CSharp ## CSharp async： ## .Net Core ## InPage Search ## My Notes * Jupyter: \[[Source](https://github.com/fzinfz/scripts/tree/master/jupyter)] * Scripts of Powershell/Python/etc: # AHK * [V2](#v2) * [Keys](#keys) * [Functions](#functions) * [SendKeys](#sendkeys) * [JoyStick](#joystick) * [Debug](#debug) * [x360](#x360) * [Keyboard](#keyboard) * [Mouse](#mouse) ## V2 v2.0.0 was released on 2022-12-20: ## Keys | Key | Name | | --- | ----- | | ^ | Ctrl | | ! | Alt | | + | Shift | | # | Win | | {+} | + | ## Functions * `SetTimer Function, Period(ms), Priority` | repeatedly/asynchronously | * Period > 0: default 250 * Period < 0: run only once * Period = 0: deleted after the thread finishes ### SendKeys ``` ^3::{ SendText "Double quote: `"" SendText 'Single quote: `'' } ``` ## JoyStick * Joy1 through Joy32 * used with `GetKeyState`: JoyX, JoyY, and JoyZ: The X (horizontal), Y (vertical), and Z (altitude/depth) axes of the controller. JoyR: The rudder(舵) or 4th axis of the controller. JoyU and JoyV: The 5th and 6th axes of the controller. JoyPOV: The point-of-view (hat) control. JoyName: The name of the controller or its driver. JoyButtons: The number of buttons supported by the controller (not always accurate). JoyAxes: The number of axes supported by the controller. JoyInfo: Example string: ZRUVPD - Z (has Z axis), R (has R axis), U (has U axis), V (has V axis) - P (has POV control), - D (the POV control has a limited number of discrete/distinct settings) - C (the POV control is continuous/fine). ### Debug * Windows Start keyword : 'joy' * .ahk : ### x360 | x360 | Default | Range | | ------------------------- | ------- | -------------------------- | | Left Stick / XY | 50 | Left/Up 0 - Right/Down 100 | | D-Pad/hat / POV | -1 | POV0 - POV31500 | | Right Stick / XY Rotation | U50 R50 | 0-100 | | ABXY | 1234 | | | LB/LeftBumper RB | 5 6 | | | Back ; Start | 7 8 | | | LT/LeftTrigger Z | Z50 | - Z100 | | RT/RightTrigger Z | Z50 | - Z0 | ## Keyboard ``` Joy2::Send "{w}" ``` ## Mouse ``` Joy1:: { Send "{LButton down}" ; Hold down the left mouse button. ; SetTimer WaitForButtonUp3, 10 } ``` # BI * [Sample data](#sample-data) * [Diagram image generator online](#diagram-image-generator-online) * [view & analyze Pandas](#view--analyze-pandas) * [plotly - for Math](#plotly---for-math) * [plot time series data](#plot-time-series-data) * [Math graph](#math-graph) * [bokeh- Python](#bokeh--python) * [HoloViews](#holoviews) * [Superset vs Redash vs Metabase](#superset-vs-redash-vs-metabase) * [Apache Superset (Caravel/Panoramix)](#apache-superset-caravelpanoramix) * [Redash](#redash) * [ELK](#elk) * [My `docker run` scripts](#my-docker-run-scripts) * [Snapshot And Restore](#snapshot-and-restore) * [Backup](#backup) * [Check](#check) * [ETL](#etl) * [kettle -JAVA](#kettle--java) * [Excel](#excel) * [xlwings](#xlwings) * [django](#django) * [.Net](#net) * [Word](#word) * [highcharts](#highcharts) ## Sample data python: ## Diagram image generator online ## view & analyze Pandas ## plotly - for Math ## plot time series data ## Math graph ## bokeh- Python Interactive Web Plotting ## HoloViews annotate your data and let it visualize itself ## Superset vs Redash vs Metabase ## Apache Superset (Caravel/Panoramix) ![](https://cloud.githubusercontent.com/assets/130878/20371438/a703a2a0-ac19-11e6-80c4-00a47c2eb644.gif) ## Redash ![](https://cloud.githubusercontent.com/assets/71468/17391289/8e83878e-5a1d-11e6-8938-af9054a33b19.gif) ## ELK ### My `docker run` scripts ### Snapshot And Restore #### Backup ``` PUT /_snapshot/bak_1 { "type": "fs", "settings": { "compress" : true, "location": "/usr/share/elasticsearch/data/backup" } } PUT /_snapshot/bak_1/snapshot_1?wait_for_completion=true ``` #### Check ``` GET /_snapshot/bak_1 ``` ## ETL ### kettle -JAVA ## Excel ### xlwings ### django django-excel: django-import-export: ### .Net ## Word ## highcharts Free for non-commercial # LBS * [Standard](#standard) * [DB](#db) * [PostGIS for PostgreSQL](#postgis-for-postgresql) * [GeoCouch for CouchDB](#geocouch-for-couchdb) * [GeoIP2](#geoip2) * [Solr](#solr) * [AMAQ](#amaq) * [Tencent](#tencent) ## Standard The GeoJSON Format: ## DB ### PostGIS for PostgreSQL ### GeoCouch for CouchDB ## GeoIP2 ## Solr ## AMAQ ## Tencent # android * [emulator](#emulator) ## emulator ``` emulator -accel-check ``` # editor * [Pritable](#pritable) * [vscode](#vscode) * [vi/vim](#vivim) * [nano](#nano) * [WebStorm](#webstorm) * [Terminal](#terminal) ## Pritable ``` pandoc -f html -t markdown # HTML to markdown pandoc test1.md -f markdown -t html -s -o test1.html # markdown to HTML # -s option says to create a “standalone” file, with a header and footer, not just a fragment. ``` Chrome/Edge - print .md on Github: ## vscode [/ms/vscode.md](/ms/vscode) ## vi/vim ``` go to line: Esc , line#, Shift-g :%s/pattern/replace/g_ # i/I: case in/sensitive cw => change word ciw => change word from cursor :w !sudo tee % ===> sudo save ~/.vimrc set nocompatible # fix array not working in insert mode ``` * Visual Insert Mode paste: Shift+Insert * VISUAL BLOCK mode Ctrl+V；【select block】；Shift + I; 【type text】; Esc ## nano CTRL + Shift + 6: mark CTRL + K: cut/delete ## WebStorm ### Terminal * Support ctrl+C/V , clickable links * env != OS env: Settings | Tools | Terminal # flutter\_web ## Start ``` flutter channel stable flutter upgrade flutter devices flutter doctor flutter create myapp cd myapp flutter run # -d chrome flutter run --release # --web-renderer html or --web-renderer canvaskit ``` ## Web renderers * HTML renderer\ Uses a combination of HTML elements, CSS, Canvas elements, and SVG elements. This renderer has a smaller download size. * CanvasKit renderer\ This renderer is fully consistent with Flutter mobile and desktop, has faster performance with higher widget density, but adds about 2MB in download size. ## adaptive vs responsive * Responsive\ Typically, a responsive app has had its layout tuned for the available screen size. Often this means (for example), re-laying out the UI if the user resizes the window, or changes the device’s orientation. This is especially necessary when the same app can run on a variety of devices, from a watch, phone, tablet, to a laptop or desktop computer. * Adaptive\ Adapting an app to run on different device types, such as mobile and desktop, requires dealing with mouse and keyboard input, as well as touch input. It also means there are different expectations about the app’s visual density, how component selection works (cascading menus vs bottom sheets, for example), using platform-specific features (such as top-level windows), and more. # git * [Learn](#learn) * [Workflow](#workflow) * [commit](#commit) * [log](#log) * [branch and merge](#branch-and-merge) * [Undo](#undo) * [stashing](#stashing) * [remote and push](#remote-and-push) * [delete remote branch](#delete-remote-branch) * [config](#config) * [Proxy](#proxy) * [filter-branch](#filter-branch) * [Remove File](#remove-file) * [Unpushed commit](#unpushed-commit) * [Every commit](#every-commit) * [from Github](#from-github) * [rebase](#rebase) * [fork](#fork) * [Submodules](#submodules) * [Github API](#github-api) * [Download by tag](#download-by-tag) * [curl Github](#curl-github) * [Query latest release](#query-latest-release) * [Github](#github) * [Self-host git servers](#self-host-git-servers) * [Tools](#tools) * [Tools](#tools-1) ## Learn * Practise: * Cheatsheet * 4 Pages: * 2 Pages: * 2 Pages: * Books * [my functions](https://github.com/fzinfz/scripts/blob/master/lib/git.sh#L1): ```bash source /dev/stdin <<< "$(curl -sSL https://raw.githubusercontent.com/fzinfz/scripts/master/lib/git.sh)" ``` ## Workflow ![](https://i.stack.imgur.com/caci5.png) ``` cat .git/HEAD # current branch head ref: refs/heads/master cat .git/ORIG_HEAD ; git show | head -1 ``` `working tree` don't include `untracked files` ### commit ``` git commit -am "save arezzo files" git commit --amend -c, --reedit-message # reuse and edit message from specified commit -C, --reuse-message # reuse message from specified commit ``` ### log ``` git log [] [] [[--] ...] git show [] ... git show [path] # details of last commit log git log --since=2.weeks ``` ### branch and merge ``` git branch -a # list all git checkout -b dev # create and checkout a new branch git checkout master git merge dev # merge dev into master --squash create a single commit instead of doing a merge --abort abort the current in-progress merge ``` ### Undo ``` git checkout -- # discard changes in working directory git reset HEAD~2 # undo 2 changes that haven’t been shared git reset # opposite of `git add ` --soft reset HEAD only --mixed reset HEAD and index (default) --hard reset HEAD, index and working tree --merge reset HEAD, index and working tree --keep reset HEAD but keep local changes git revert HEAD~2 # undo 2 changes on a public branch ``` ``` git reset master git checkout master ``` ![](https://i.stack.imgur.com/UWGiw.png) ### stashing ``` git stash git stash list git stash apply/pop git stash apply stash@{2} git stash drop git stash drop stash@{0} ``` ### remote and push ``` git remote -v git remote add [] -f, --fetch fetch the remote branches --tags --no-tags -t, --track # branch(es) to track -m, --master --mirror[=] # set up remote as a mirro cat .git/config `refspec`: +: is the pattern for references on the remote side is where those references will be tracked locally [remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* fetch = +refs/heads/master:refs/remotes/origin/master fetch = +refs/heads/foo/*:refs/remotes/origin/foo/* git push -u origin master:refs/heads/foo/master push = refs/heads/master:refs/heads/qa/master tree .git/refs/ .git/refs/ ├── heads │   ├── dev │   └── master ├── remotes │   └── origin │   └── HEAD └── tags git push [] [ [...]] -u, --set-upstream set upstream for git pull/status git pull origin master # fetch + merge ``` #### delete remote branch ``` git push origin :topic git push origin --delete topic ``` ## config ``` git config -l --global ~/.gitconfig --local .git/config core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true --system /etc/gitconfig --get get value: name [value-regex] --get-all get all values: key [value-regex] --get-regexp get values for regexp: name-regex [value-regex] --get-urlmatch get value specific for the URL: section[.var] URL --replace-all replace all matching variables: name value [value_regex] --add add a new variable: name value --unset remove a variable: name [value-regex] --unset-all remove all matches: name [value-regex] git config --global push.default simple git config --list --show-origin ``` ### Proxy ``` git config --global http.proxy http://$IP:$Port ``` ## filter-branch ``` [--setup ] [--env-filter ] [--tree-filter ] [--index-filter ] [--parent-filter ] [--msg-filter ] [--commit-filter ] [--tag-name-filter ] [--subdirectory-filter ] [--original ] [-d ] [-f | --force] [--] [...] ``` ## Remove File ### Unpushed commit ``` git rm --cached giant_file # leave it on disk git commit --amend -CHEAD ``` ### Every commit ``` git filter-branch --tree-filter 'rm -f filename' HEAD --all # all branches git filter-branch --index-filter \ 'git rm --cached --ignore-unmatch filename' HEAD ``` ### from Github ## rebase ``` git remote add upstream https://github.com/yeasy/docker_practice git fetch upstream git checkout master git rebase upstream/master git push -f origin master ``` ## fork ## Submodules treat the two projects as separate yet still be able to use one from within the other. ## Github API ### Download by tag ``` curl -sSL https://api.github.com/repos/django/django/tags \ | jq '.[0].zipball_url' | xargs -t wget -O file.zip ``` ### curl Github ``` curl -H 'Authorization: token INSERT_ACCESS_TOKEN_HERE' \ -H 'Accept: application/vnd.github.v3.raw' -O -L \ https://api.github.com/repos/owner/repo/contents/path ``` ### Query latest release ``` curl -sSL https://api.github.com/repos/ParsePlatform/Parse-SDK-Android/releases/latest \ | jq '.zipball_url' | xargs -t wget -O file.zip ``` ## Github check watchers: ## Self-host git servers Go: Go: Ruby: ## Tools curl -L .html to page, etc: .js CDN: .ipynb fast open: ## Tools Code search: download: # go * [Articles](#articles) * [Go Environment variables](#go-environment-variables) ## Articles (Chinese) ## Go Environment variables ``` $GOOS $GOARCH android arm darwin 386 darwin amd64 darwin arm darwin arm64 dragonfly amd64 freebsd 386 freebsd amd64 freebsd arm linux 386 linux amd64 linux arm linux arm64 linux ppc64 linux ppc64le linux mips linux mipsle linux mips64 linux mips64le netbsd 386 netbsd amd64 netbsd arm openbsd 386 openbsd amd64 openbsd arm plan9 386 plan9 amd64 solaris amd64 windows 386 windows amd64 ``` ## Check ``` dpkg -l golang-{go,src} | grep ^ii ``` # HTML5/BS * [Chrome](#chrome) * [Percent-encoding reserved](#percent-encoding-reserved) * [HTML5](#html5) * [lang](#lang) * [Subresource Integrity](#subresource-integrity) * [media type](#media-type) * [icons](#icons) * [Events](#events) * [CSS box model](#css-box-model) * [box-sizing](#box-sizing) * [content](#content) * [padding](#padding) * [border](#border) * [margin](#margin) * [Spacing](#spacing) * [Mobile](#mobile) * [Flexbox](#flexbox) * [UI Frameworks](#ui-frameworks) * [Bootstrap](#bootstrap) * [Version 3](#version-3) * [Version 4](#version-4) * [Version 5](#version-5) * [Grid system](#grid-system) * [forms](#forms) * [whitespace and wrap](#whitespace-and-wrap) * [robots](#robots) * [Frameworks](#frameworks) ## Chrome ``` hard reload: Ctrl + F5 F12 -> Ctrl+Shift+P -> "screenshot" delete address history: Shift + Del remove domain name: F12 -> console document.body.innerHTML = document.body.innerHTML.replace(/\.\w+\.\w+(?=:)/g, "") ``` ## Percent-encoding reserved | ! | # | $ | % | & | ' | ( | ) | \* | + | , | / | : | ; | = | ? | @ | \[ | ] | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | %21 | %23 | %24 | %25 | %26 | %27 | %28 | %29 | %2A | %2B | %2C | %2F | %3A | %3B | %3D | %3F | %40 | %5B | %5D | ## HTML5 ### lang empty string indicates that the primary language is unknown ### Subresource Integrity ## media type MIME type or content type: ``` application/pdf audio font example image message model multipart text video ``` ## icons ## Events ``` cut/copy/paste input: the value of an ,